2023-02-23T09:51:59.600 INFO:root:teuthology version: 0.0.1.dev41+gfbbadb5
2023-02-23T09:51:59.736 INFO:teuthology.lock.ops:Start node 'smithi090.front.sepia.ceph.com' reimaging
2023-02-23T09:51:59.736 INFO:teuthology.lock.ops:Updating [smithi090.front.sepia.ceph.com]: reset os type and version on server
2023-02-23T09:51:59.737 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server
2023-02-23T09:51:59.761 INFO:teuthology.lock.ops:Node 'smithi090.front.sepia.ceph.com' reimaging is complete
2023-02-23T09:51:59.762 INFO:teuthology.lock.ops:Start node 'smithi173.front.sepia.ceph.com' reimaging
2023-02-23T09:51:59.762 INFO:teuthology.lock.ops:Updating [smithi173.front.sepia.ceph.com]: reset os type and version on server
2023-02-23T09:51:59.762 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server
2023-02-23T09:51:59.805 INFO:teuthology.lock.ops:Node 'smithi173.front.sepia.ceph.com' reimaging is complete
2023-02-23T09:51:59.923 INFO:teuthology.provision.fog.smithi090:Scheduling deploy of centos 8.stream
2023-02-23T09:51:59.961 INFO:teuthology.provision.fog.smithi173:Scheduling deploy of centos 8.stream
2023-02-23T09:52:00.117 INFO:teuthology.orchestra.console:Power off smithi090
2023-02-23T09:52:00.158 INFO:teuthology.orchestra.console:Power off smithi173
2023-02-23T09:52:00.271 INFO:teuthology.orchestra.console:Power off for smithi090 completed
2023-02-23T09:52:00.307 INFO:teuthology.orchestra.console:Power off for smithi173 completed
2023-02-23T09:52:00.373 INFO:teuthology.orchestra.console:Power on smithi090
2023-02-23T09:52:00.408 INFO:teuthology.orchestra.console:Power on smithi173
2023-02-23T09:52:04.664 INFO:teuthology.orchestra.console:Power on for smithi173 completed
2023-02-23T09:52:04.765 INFO:teuthology.provision.fog.smithi173:Waiting for deploy to finish
2023-02-23T09:52:08.747 INFO:teuthology.orchestra.console:Power on for smithi090 completed
2023-02-23T09:52:08.847 INFO:teuthology.provision.fog.smithi090:Waiting for deploy to finish
2023-02-23T09:54:53.637 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-02-23T09:55:02.693 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-02-23T09:55:11.781 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-02-23T09:55:20.837 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-02-23T09:55:27.781 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-02-23T09:55:29.893 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-02-23T09:55:36.837 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-02-23T09:55:38.949 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.173
2023-02-23T09:55:45.893 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-02-23T09:55:54.949 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-02-23T09:56:04.005 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-02-23T09:56:13.061 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-02-23T09:56:22.117 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-02-23T09:56:44.969 WARNING:teuthology.provision.fog:timed out
2023-02-23T09:56:51.327 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-02-23T09:56:51.573 INFO:teuthology.provision.fog.smithi173:Node is ready
2023-02-23T09:56:51.630 INFO:teuthology.orchestra.run.smithi173.stdout:smithi173.front.sepia.ceph.com
2023-02-23T09:56:51.688 INFO:teuthology.orchestra.run.smithi173.stdout:172.21.15.173 smithi173.front.sepia.ceph.com smithi173
2023-02-23T09:56:52.034 INFO:teuthology.provision.fog.smithi173:Deploy complete!
2023-02-23T09:57:28.157 WARNING:teuthology.provision.fog:timed out
2023-02-23T09:57:34.512 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-02-23T09:57:34.866 INFO:teuthology.provision.fog.smithi090:Node is ready
2023-02-23T09:57:34.884 INFO:teuthology.orchestra.run.smithi090.stdout:smithi090.front.sepia.ceph.com
2023-02-23T09:57:34.942 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90 smithi090.front.sepia.ceph.com smithi090
2023-02-23T09:57:35.307 INFO:teuthology.provision.fog.smithi090:Deploy complete!
2023-02-23T09:57:35.464 INFO:teuthology.lock.ops:Checking smithi090.front.sepia.ceph.com
2023-02-23T09:57:35.464 INFO:teuthology.lock.ops:New key found. Updating...
2023-02-23T09:57:35.491 INFO:teuthology.lock.ops:Checking smithi173.front.sepia.ceph.com
2023-02-23T09:57:35.491 INFO:teuthology.lock.ops:New key found. Updating...
2023-02-23T09:58:20.861 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/dispatcher/supervisor.py", line 209, in reimage
    reimaged = reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/lock/ops.py", line 289, in do_update_keys
    return push_new_keys(keys_dict, reference), keys_dict
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/lock/ops.py", line 298, in push_new_keys
    if not update_lock(hostname, ssh_pub_key=pubkey):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/lock/ops.py", line 242, in update_lock
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: 'update lock smithi173.front.sepia.ceph.com' reached maximum tries (10) after waiting for 54.69909370269514 seconds
2023-02-23T09:58:20.925 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-22_21:31:50-rados-wip-yuri10-testing-2023-02-22-0848-distro-default-smithi/jobs/ failed with status 500: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>
    
  </body>
</html>

2023-02-23T09:58:21.979 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-22_21:31:50-rados-wip-yuri10-testing-2023-02-22-0848-distro-default-smithi/jobs/ failed with status 500: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>
    
  </body>
</html>

2023-02-23T09:58:23.486 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-22_21:31:50-rados-wip-yuri10-testing-2023-02-22-0848-distro-default-smithi/jobs/ failed with status 500: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>
    
  </body>
</html>

2023-02-23T09:58:25.446 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-22_21:31:50-rados-wip-yuri10-testing-2023-02-22-0848-distro-default-smithi/jobs/ failed with status 500: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>
    
  </body>
</html>

2023-02-23T09:58:27.854 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-22_21:31:50-rados-wip-yuri10-testing-2023-02-22-0848-distro-default-smithi/jobs/ failed with status 500: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>
    
  </body>
</html>

2023-02-23T09:58:30.718 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-22_21:31:50-rados-wip-yuri10-testing-2023-02-22-0848-distro-default-smithi/jobs/ failed with status 500: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>
    
  </body>
</html>

2023-02-23T09:58:34.026 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-22_21:31:50-rados-wip-yuri10-testing-2023-02-22-0848-distro-default-smithi/jobs/ failed with status 500: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>
    
  </body>
</html>

2023-02-23T09:58:37.790 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-22_21:31:50-rados-wip-yuri10-testing-2023-02-22-0848-distro-default-smithi/jobs/ failed with status 500: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>
    
  </body>
</html>

2023-02-23T09:58:42.001 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-22_21:31:50-rados-wip-yuri10-testing-2023-02-22-0848-distro-default-smithi/jobs/ failed with status 500: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>
    
  </body>
</html>

2023-02-23T09:58:46.662 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-22_21:31:50-rados-wip-yuri10-testing-2023-02-22-0848-distro-default-smithi/jobs/ failed with status 500: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>
    
  </body>
</html>

2023-02-23T09:58:46.663 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/dispatcher/supervisor.py", line 209, in reimage
    reimaged = reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged = do_update_keys(list(reimaged.keys()))[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/lock/ops.py", line 289, in do_update_keys
    return push_new_keys(keys_dict, reference), keys_dict
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/lock/ops.py", line 298, in push_new_keys
    if not update_lock(hostname, ssh_pub_key=pubkey):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/lock/ops.py", line 242, in update_lock
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: 'update lock smithi173.front.sepia.ceph.com' reached maximum tries (10) after waiting for 54.69909370269514 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/dispatcher/__init__.py", line 63, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/dispatcher/supervisor.py", line 52, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/dispatcher/supervisor.py", line 213, in reimage
    report.try_push_job_info(ctx.config, dict(status='dead', failure_reason='Error reimaging machines: ' + str(e)))
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/report.py", line 508, in try_push_job_info
    push_job_info(run_name, job_id, job_info)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/report.py", line 473, in push_job_info
    reporter.report_job(run_name, job_id, job_info)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/report.py", line 298, in report_job
    while proceed():
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: 'report job 7184839' reached maximum tries (10) after waiting for 30.27453816323822 seconds