2023-09-21T11:31:13.581 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-21T11:31:13.583 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T11:31:13.717 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f3969213ca0>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f3968fb9b80>, signals=[15])
2023-09-21T11:31:13.718 INFO:teuthology.lock.ops:Start node 'smithi099.front.sepia.ceph.com' reimaging
2023-09-21T11:31:13.719 INFO:teuthology.lock.ops:Updating [smithi099.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T11:31:13.719 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server
2023-09-21T11:31:13.742 INFO:teuthology.lock.ops:Start node 'smithi187.front.sepia.ceph.com' reimaging
2023-09-21T11:31:13.743 INFO:teuthology.lock.ops:Updating [smithi187.front.sepia.ceph.com]: reset os type and version on server
2023-09-21T11:31:13.743 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server
2023-09-21T11:31:13.744 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /host {"name": "smithi099"}
2023-09-21T11:31:13.767 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /host {"name": "smithi187"}
2023-09-21T11:31:13.804 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /image {"name": "smithi_ubuntu_20.04"}
2023-09-21T11:31:13.826 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /image {"name": "smithi_ubuntu_20.04"}
2023-09-21T11:31:13.846 DEBUG:teuthology.provision.fog.smithi099:Requesting image smithi_ubuntu_20.04 (ID 2)
2023-09-21T11:31:13.846 DEBUG:teuthology.provision.fog.smithi099:FOG request: PUT /host/204 {"imageID": 2}
2023-09-21T11:31:13.866 DEBUG:teuthology.provision.fog.smithi187:Requesting image smithi_ubuntu_20.04 (ID 2)
2023-09-21T11:31:13.866 DEBUG:teuthology.provision.fog.smithi187:FOG request: PUT /host/292 {"imageID": 2}
2023-09-21T11:31:13.887 INFO:teuthology.provision.fog.smithi099:Scheduling deploy of ubuntu 20.04
2023-09-21T11:31:13.887 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /task/active None
2023-09-21T11:31:13.921 INFO:teuthology.provision.fog.smithi187:Scheduling deploy of ubuntu 20.04
2023-09-21T11:31:13.921 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /task/active None
2023-09-21T11:31:13.953 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T11:31:13.991 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /tasktype {"name": "deploy"}
2023-09-21T11:31:13.994 DEBUG:teuthology.provision.fog.smithi099:FOG request: POST /host/204/task {"taskTypeID": 1}
2023-09-21T11:31:14.032 DEBUG:teuthology.provision.fog.smithi187:FOG request: POST /host/292/task {"taskTypeID": 1}
2023-09-21T11:31:14.043 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /task/active None
2023-09-21T11:31:14.078 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /task/active None
2023-09-21T11:31:14.153 INFO:teuthology.orchestra.console.smithi099:Power off
2023-09-21T11:31:14.153 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:31:14.186 INFO:teuthology.orchestra.console.smithi187:Power off
2023-09-21T11:31:14.186 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:31:14.207 DEBUG:teuthology.orchestra.console.smithi187:power off output: Chassis Power Control: Down/Off
2023-09-21T11:31:14.208 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:31:14.230 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is off
2023-09-21T11:31:14.331 INFO:teuthology.orchestra.console.smithi187:Power off completed
2023-09-21T11:31:14.432 INFO:teuthology.orchestra.console.smithi187:Power on
2023-09-21T11:31:14.432 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T11:31:14.451 DEBUG:teuthology.orchestra.console.smithi187:power on output: Chassis Power Control: Up/On
2023-09-21T11:31:14.452 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:31:14.474 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is off
2023-09-21T11:31:18.479 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:31:18.587 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is off
2023-09-21T11:31:22.592 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:31:22.766 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on
2023-09-21T11:31:22.867 INFO:teuthology.orchestra.console.smithi187:Power on completed
2023-09-21T11:31:22.967 INFO:teuthology.provision.fog.smithi187:Waiting for deploy to finish
2023-09-21T11:31:22.968 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /task/active None
2023-09-21T11:31:34.223 DEBUG:teuthology.orchestra.console.smithi099:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:31:34.224 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:31:38.061 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /task/active None
2023-09-21T11:31:53.155 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /task/active None
2023-09-21T11:31:54.295 DEBUG:teuthology.orchestra.console.smithi099:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:31:54.295 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:32:08.262 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /task/active None
2023-09-21T11:32:14.369 DEBUG:teuthology.orchestra.console.smithi099:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:32:14.370 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:32:15.378 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:32:19.383 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:32:20.492 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:32:23.350 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /task/active None
2023-09-21T11:32:24.495 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:32:25.603 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:32:29.608 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:32:30.717 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:32:34.722 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:32:35.831 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:32:38.429 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /task/active None
2023-09-21T11:32:39.834 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:32:40.944 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:32:44.946 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:32:46.054 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:32:50.059 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:32:51.168 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:32:53.521 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /task/active None
2023-09-21T11:32:55.170 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:32:56.279 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:33:00.282 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:33:01.390 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:33:05.394 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:33:06.502 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:33:08.610 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /task/active None
2023-09-21T11:33:10.504 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:33:11.613 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:33:15.618 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:33:16.726 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:33:20.730 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:33:21.838 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:33:23.714 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /task/active None
2023-09-21T11:33:25.841 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:33:26.950 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:33:26.951 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T11:33:26.951 ERROR:teuthology.orchestra.console.smithi099:Failed to power off
2023-09-21T11:33:26.951 INFO:teuthology.orchestra.console.smithi099:Power on
2023-09-21T11:33:26.952 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T11:33:38.810 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /task/active None
2023-09-21T11:33:47.021 DEBUG:teuthology.orchestra.console.smithi099:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:33:47.021 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T11:33:53.905 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /task/active None
2023-09-21T11:34:07.089 DEBUG:teuthology.orchestra.console.smithi099:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:34:07.089 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-09-21T11:34:09.015 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /task/active None
2023-09-21T11:34:24.262 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /task/active None
2023-09-21T11:34:27.157 DEBUG:teuthology.orchestra.console.smithi099:power on output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:34:27.157 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:34:28.166 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:34:32.170 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:34:33.279 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:34:37.282 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:34:38.390 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:34:39.357 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /task/active None
2023-09-21T11:34:42.395 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:34:43.503 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:34:47.506 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:34:48.615 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:34:52.618 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:34:53.727 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:34:54.438 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /task/active None
2023-09-21T11:34:57.730 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:34:58.838 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:35:02.842 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:35:03.950 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:35:07.954 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:35:09.063 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:35:09.516 DEBUG:teuthology.provision.fog.smithi187:FOG request: GET /task/active None
2023-09-21T11:35:09.595 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:35:13.066 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:35:14.174 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:35:18.178 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:35:19.287 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:35:23.290 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:35:24.398 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:35:28.154 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-09-21T11:35:28.399 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:35:29.508 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:35:33.510 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:35:34.157 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:35:34.619 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:35:37.210 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-09-21T11:35:38.621 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:35:39.730 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:35:39.731 WARNING:teuthology.contextutil:'wait for power on' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T11:35:39.731 DEBUG:teuthology.provision.fog.smithi099:Canceling deploy task with ID 499319
2023-09-21T11:35:39.731 DEBUG:teuthology.provision.fog.smithi099:FOG request: DELETE /task/cancel {"id": 499319}
2023-09-21T11:35:39.789 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi099
2023-09-21T11:35:39.790 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-09-21T11:35:39.928 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=58612b4dadc94830a820ad98649e4418
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi099
2023-09-21T11:35:39.930 INFO:teuthology.nuke:Checking targets against current locks
2023-09-21T11:35:39.990 DEBUG:teuthology.nuke:shortname: smithi099
2023-09-21T11:35:39.991 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T11:35:40.015 DEBUG:teuthology.nuke:shortname: smithi187
2023-09-21T11:35:40.015 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-09-21T11:35:40.019 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399786', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 11:31:12.795420', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFXRHbCQKDEuyNLFw4V1AtfgTXVj1Soceqgh+qxIZuwuU9XS7E6mMjA4WiDriC/6XhxlgPBiXLgPCx7jyVR8EVs='}
2023-09-21T11:35:40.037 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-09-20_16:32:45-fs-wip-lusov-config-format-distro-default-smithi/7399786', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-21 11:31:12.797227', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJCHW6YqhQcTBspmWknuFFI+p+YJfSBstrv4TTkOJyxoQ3OHoT7eYUH5F3XaZdo7xu4hEyWx3IlxW2jZ0shlKgI='}
2023-09-21T11:35:40.063 INFO:teuthology.orchestra.console.smithi099:Power off
2023-09-21T11:35:40.064 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:35:40.102 INFO:teuthology.orchestra.console.smithi187:Power off
2023-09-21T11:35:40.103 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:35:40.126 DEBUG:teuthology.orchestra.console.smithi187:power off output: Chassis Power Control: Down/Off
2023-09-21T11:35:40.126 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:35:40.151 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on
2023-09-21T11:35:43.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:35:44.154 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:35:44.262 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on
2023-09-21T11:35:46.266 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-09-21T11:35:48.268 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:35:48.377 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is off
2023-09-21T11:35:48.478 INFO:teuthology.orchestra.console.smithi187:Power off completed
2023-09-21T11:35:48.636 INFO:teuthology.lock.ops:unlocked: smithi187.front.sepia.ceph.com
2023-09-21T11:35:52.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:35:55.326 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-09-21T11:36:00.131 DEBUG:teuthology.orchestra.console.smithi099:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:36:00.132 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:36:01.330 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:36:04.410 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-09-21T11:36:10.414 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:36:13.470 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-09-21T11:36:19.476 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:36:20.200 DEBUG:teuthology.orchestra.console.smithi099:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:36:20.200 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-09-21T11:36:22.558 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-09-21T11:36:28.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:36:31.642 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-09-21T11:36:37.646 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:36:40.271 DEBUG:teuthology.orchestra.console.smithi099:power off output: Error: Unable to establish IPMI v2 / RMCP+ session
2023-09-21T11:36:40.272 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:36:40.698 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-09-21T11:36:41.281 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:36:45.283 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:36:46.392 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:36:46.701 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:36:49.758 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-09-21T11:36:50.394 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:36:51.503 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:36:55.506 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:36:55.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:36:56.617 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:36:58.842 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-09-21T11:37:00.621 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:37:01.730 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:37:04.846 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:37:05.732 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:37:06.842 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:37:07.902 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-09-21T11:37:10.846 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:37:11.956 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:37:13.906 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:37:15.960 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:37:17.069 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:37:21.074 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:37:22.184 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:37:26.188 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:37:27.298 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:37:31.302 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:37:32.314 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-09-21T11:37:32.410 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:37:36.417 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:37:37.527 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:37:38.318 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:37:41.374 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-09-21T11:37:41.529 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:37:42.638 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:37:46.642 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:37:47.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-09-21T11:37:47.505 ERROR:teuthology.orchestra.connection:Error authenticating with smithi187.front.sepia.ceph.com: Authentication failed.
2023-09-21T11:37:47.509 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-09-21T11:37:47.510 DEBUG:teuthology.orchestra.run.smithi187:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-09-21T11:37:47.513 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T11:37:47.750 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:37:48.519 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T11:37:49.526 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T11:37:50.532 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T11:37:51.538 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T11:37:51.752 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-09-21T11:37:52.545 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-09-21T11:37:52.862 DEBUG:teuthology.orchestra.console.smithi099:check power output: 
2023-09-21T11:37:52.863 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds
2023-09-21T11:37:52.863 ERROR:teuthology.orchestra.console.smithi099:Failed to power off
2023-09-21T11:37:53.005 INFO:teuthology.lock.ops:unlocked: smithi099.front.sepia.ceph.com
2023-09-21T11:37:53.233 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/dispatcher/supervisor.py", line 228, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/provision/fog.py", line 85, in create
    self.remote.console.power_on()
  File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/console.py", line 252, in power_on
    raise RuntimeError(err_msg)
RuntimeError: Failed to power on smithi099