2023-06-28T16:30:51.566 INFO:root:teuthology version: 0.0.1.dev132+gaa9345a 2023-06-28T16:30:51.567 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T16:30:51.568 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T16:30:51.727 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f881dd8b3a0>, signals=[15]) 2023-06-28T16:30:51.728 INFO:teuthology.lock.ops:Start node 'smithi006.front.sepia.ceph.com' reimaging 2023-06-28T16:30:51.729 INFO:teuthology.lock.ops:Updating [smithi006.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T16:30:51.729 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2023-06-28T16:30:51.762 INFO:teuthology.lock.ops:Start node 'smithi132.front.sepia.ceph.com' reimaging 2023-06-28T16:30:51.762 INFO:teuthology.lock.ops:Updating [smithi132.front.sepia.ceph.com]: reset os type and version on server 2023-06-28T16:30:51.762 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2023-06-28T16:30:51.764 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /host {"name": "smithi006"} 2023-06-28T16:30:54.911 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /host {"name": "smithi132"} 2023-06-28T16:30:59.738 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /image {"name": "smithi_rhel_8.4"} 2023-06-28T16:31:03.748 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /image {"name": "smithi_rhel_8.4"} 2023-06-28T16:31:03.808 DEBUG:teuthology.provision.fog.smithi006:Requesting image smithi_rhel_8.4 (ID 7) 2023-06-28T16:31:03.809 DEBUG:teuthology.provision.fog.smithi006:FOG request: PUT /host/112 {"imageID": 7} 2023-06-28T16:31:03.811 DEBUG:teuthology.provision.fog.smithi132:Requesting image smithi_rhel_8.4 (ID 7) 2023-06-28T16:31:03.811 DEBUG:teuthology.provision.fog.smithi132:FOG request: PUT /host/237 {"imageID": 7} 2023-06-28T16:31:03.858 INFO:teuthology.provision.fog.smithi132:Scheduling deploy of rhel 8.4 2023-06-28T16:31:03.859 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T16:31:03.867 INFO:teuthology.provision.fog.smithi006:Scheduling deploy of rhel 8.4 2023-06-28T16:31:03.867 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-06-28T16:31:03.945 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T16:31:03.951 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /tasktype {"name": "deploy"} 2023-06-28T16:31:03.991 DEBUG:teuthology.provision.fog.smithi132:FOG request: POST /host/237/task {"taskTypeID": 1} 2023-06-28T16:31:03.997 DEBUG:teuthology.provision.fog.smithi006:FOG request: POST /host/112/task {"taskTypeID": 1} 2023-06-28T16:31:04.056 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T16:31:04.063 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-06-28T16:31:04.181 INFO:teuthology.orchestra.console:Power off smithi132 2023-06-28T16:31:04.182 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T16:31:04.199 INFO:teuthology.orchestra.console:Power off smithi006 2023-06-28T16:31:04.199 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T16:31:04.210 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:04.223 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:04.333 INFO:teuthology.orchestra.console:Power off for smithi132 completed 2023-06-28T16:31:04.345 INFO:teuthology.orchestra.console:Power off for smithi006 completed 2023-06-28T16:31:04.435 INFO:teuthology.orchestra.console:Power on smithi132 2023-06-28T16:31:04.435 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T16:31:04.446 INFO:teuthology.orchestra.console:Power on smithi006 2023-06-28T16:31:04.446 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-28T16:31:04.459 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:04.468 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:08.484 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:08.494 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:12.601 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:12.610 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:17.713 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:17.721 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:22.824 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:22.834 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:27.941 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:27.949 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:33.053 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:33.063 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:38.167 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:38.175 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:43.279 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:43.288 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:43.491 INFO:teuthology.orchestra.console:Power on for smithi132 completed 2023-06-28T16:31:43.592 INFO:teuthology.provision.fog.smithi132:Waiting for deploy to finish 2023-06-28T16:31:43.592 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T16:31:52.600 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:31:53.491 INFO:teuthology.orchestra.console:Power on for smithi006 completed 2023-06-28T16:31:53.592 INFO:teuthology.provision.fog.smithi006:Waiting for deploy to finish 2023-06-28T16:31:53.593 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-06-28T16:32:08.605 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T16:32:23.748 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-06-28T16:32:28.753 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T16:32:48.856 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T16:32:48.858 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-06-28T16:33:03.992 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T16:33:04.013 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-06-28T16:33:19.118 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T16:33:24.043 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-06-28T16:33:39.314 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T16:33:39.748 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-06-28T16:33:40.065 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:33:40.184 ERROR:teuthology.orchestra.connection:Error authenticating with smithi006.front.sepia.ceph.com: Authentication failed. 2023-06-28T16:33:40.186 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T16:33:40.187 DEBUG:teuthology.orchestra.run.smithi006:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T16:33:40.189 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-06-28T16:33:54.918 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T16:34:10.077 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T16:34:30.154 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T16:34:50.258 DEBUG:teuthology.provision.fog.smithi132:FOG request: GET /task/active None 2023-06-28T16:34:53.700 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:35:12.112 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132 2023-06-28T16:35:18.114 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:35:26.192 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132 2023-06-28T16:35:32.080 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-06-28T16:35:32.083 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer 2023-06-28T16:35:32.084 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-28T16:35:32.085 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-28T16:35:37.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:35:37.177 INFO:teuthology.nuke:Checking targets against current locks 2023-06-28T16:35:52.249 DEBUG:teuthology.nuke:shortname: smithi006 2023-06-28T16:35:52.250 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T16:35:57.256 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132 2023-06-28T16:36:02.284 DEBUG:teuthology.nuke:shortname: smithi132 2023-06-28T16:36:02.285 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-28T16:36:07.289 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:36:12.294 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-28_14:13:04-rados-quincy_release-distro-default-smithi/7320380', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 16:30:50.746524', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHvdkUZoVJZ4p7sfqOIRdNXCK0e4xIby+g0kx7+7yPzRibjyIf70ZGw0/0njNdzJ7Ew8zLb53CcNo9e908XNVekqqSakt3QNBvvpB4BzDNT4bGQyy7ytMfY/KKnpvoKsHgBevpzTDgitDDtWioMVvv8UrTwIoCFtGTyE4kP8p6JgWOnH13DuHQE982xJ3EC1xAiHqow3XdiXvEaOzEFNdI1t/Ru/ot7rqdiLORDOW5A0+RrBXc+UbXtyWHsYkz3zBEVme/t2soCUXKk0V57V3LgkL9rlAIrOEyD/oZm7a1ObGVKKspFVZtXxe4VNOu8jopO3+x7caKYcrLcr0swcMou9HRGkNUtKq7JOkA/W7VwJSn21dRQNNC7FWueBi5Cugelwd5iFDSL+VCqmGpuRrVc8+4wbun0qT6A6k9FmqoZSVMT5jHYuSKkLO3R4KZiiLXpgehd25CXrEO4tRvYwjkOjvPwkleQ9oh3YvO+8ygRu6+969YK2ok5/FF1yW/zkc='} 2023-06-28T16:36:17.299 WARNING:teuthology.provision.fog.smithi132:[Errno None] Unable to connect to port 22 on 172.21.15.132 2023-06-28T16:36:22.327 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-28_14:13:04-rados-quincy_release-distro-default-smithi/7320380', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-28 16:30:50.748728', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-06-28T16:36:24.109 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-28T16:36:29.143 INFO:teuthology.orchestra.console:Power off smithi006 2023-06-28T16:36:29.144 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T16:36:29.166 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:36:29.187 INFO:teuthology.orchestra.console:Power off smithi132 2023-06-28T16:36:29.188 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-28T16:36:29.211 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:36:29.427 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-28T16:36:29.428 DEBUG:teuthology.orchestra.run.smithi132:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-28T16:36:32.365 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-06-28T16:36:33.202 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:36:33.234 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:36:37.316 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:36:37.344 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-28T16:36:37.526 INFO:teuthology.orchestra.console:Power off for smithi006 completed 2023-06-28T16:36:37.556 INFO:teuthology.orchestra.console:Power off for smithi132 completed 2023-06-28T16:36:57.704 INFO:teuthology.lock.ops:unlocked: smithi006.front.sepia.ceph.com 2023-06-28T16:36:57.712 INFO:teuthology.lock.ops:unlocked: smithi132.front.sepia.ceph.com 2023-06-28T16:37:07.967 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 57, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/provision/fog.py", line 295, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/teuthology/orchestra/run.py", line 97, in execute self.client.exec_command(self.command, timeout=self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 510, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 919, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1050, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2109, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_aa9345a996190b15ab9d46b5969b5f8280bfba99/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) ConnectionResetError: [Errno 104] Connection reset by peer