2023-11-18T14:11:46.225 INFO:root:teuthology version: 0.0.1.dev205+g2442239 2023-11-18T14:11:46.225 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-18T14:11:46.389 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f66d698b670>, signals=[15]) 2023-11-18T14:11:46.389 INFO:teuthology.lock.ops:Start node 'smithi131.front.sepia.ceph.com' reimaging 2023-11-18T14:11:46.390 INFO:teuthology.lock.ops:Updating [smithi131.front.sepia.ceph.com]: reset os type and version on server 2023-11-18T14:11:46.390 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2023-11-18T14:11:46.410 INFO:teuthology.lock.ops:Start node 'smithi148.front.sepia.ceph.com' reimaging 2023-11-18T14:11:46.410 INFO:teuthology.lock.ops:Updating [smithi148.front.sepia.ceph.com]: reset os type and version on server 2023-11-18T14:11:46.411 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2023-11-18T14:11:46.435 INFO:teuthology.lock.ops:Start node 'smithi157.front.sepia.ceph.com' reimaging 2023-11-18T14:11:46.435 INFO:teuthology.lock.ops:Updating [smithi157.front.sepia.ceph.com]: reset os type and version on server 2023-11-18T14:11:46.435 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2023-11-18T14:11:46.533 DEBUG:teuthology.provision.fog.smithi131:Requesting image smithi_centos_9.stream (ID 10) 2023-11-18T14:11:46.539 DEBUG:teuthology.provision.fog.smithi148:Requesting image smithi_centos_9.stream (ID 10) 2023-11-18T14:11:46.571 DEBUG:teuthology.provision.fog.smithi157:Requesting image smithi_centos_9.stream (ID 10) 2023-11-18T14:11:46.581 INFO:teuthology.provision.fog.smithi148:Scheduling deploy of centos 9.stream 2023-11-18T14:11:46.584 INFO:teuthology.provision.fog.smithi131:Scheduling deploy of centos 9.stream 2023-11-18T14:11:46.617 INFO:teuthology.provision.fog.smithi157:Scheduling deploy of centos 9.stream 2023-11-18T14:11:46.785 INFO:teuthology.orchestra.console.smithi148:Power off 2023-11-18T14:11:46.785 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-18T14:11:46.797 INFO:teuthology.orchestra.console.smithi131:Power off 2023-11-18T14:11:46.798 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-18T14:11:46.808 DEBUG:teuthology.orchestra.console.smithi148:power off output: Chassis Power Control: Down/Off 2023-11-18T14:11:46.808 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:11:46.820 INFO:teuthology.orchestra.console.smithi157:Power off 2023-11-18T14:11:46.820 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-18T14:11:46.826 DEBUG:teuthology.orchestra.console.smithi131:power off output: Chassis Power Control: Down/Off 2023-11-18T14:11:46.826 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:11:46.833 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is off 2023-11-18T14:11:46.838 DEBUG:teuthology.orchestra.console.smithi157:power off output: Chassis Power Control: Down/Off 2023-11-18T14:11:46.838 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:11:46.845 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2023-11-18T14:11:46.860 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is off 2023-11-18T14:11:46.934 INFO:teuthology.orchestra.console.smithi148:Power off completed 2023-11-18T14:11:46.947 INFO:teuthology.orchestra.console.smithi131:Power off completed 2023-11-18T14:11:46.961 INFO:teuthology.orchestra.console.smithi157:Power off completed 2023-11-18T14:11:47.035 INFO:teuthology.orchestra.console.smithi148:Power on 2023-11-18T14:11:47.036 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-11-18T14:11:47.048 INFO:teuthology.orchestra.console.smithi131:Power on 2023-11-18T14:11:47.048 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-11-18T14:11:47.056 DEBUG:teuthology.orchestra.console.smithi148:power on output: Chassis Power Control: Up/On 2023-11-18T14:11:47.056 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:11:47.061 INFO:teuthology.orchestra.console.smithi157:Power on 2023-11-18T14:11:47.062 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-11-18T14:11:47.069 DEBUG:teuthology.orchestra.console.smithi131:power on output: Chassis Power Control: Up/On 2023-11-18T14:11:47.069 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:11:47.077 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is off 2023-11-18T14:11:47.083 DEBUG:teuthology.orchestra.console.smithi157:power on output: Chassis Power Control: Up/On 2023-11-18T14:11:47.083 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:11:47.090 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2023-11-18T14:11:47.102 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is off 2023-11-18T14:11:51.082 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:11:51.092 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:11:51.103 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:11:51.194 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is off 2023-11-18T14:11:51.201 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2023-11-18T14:11:51.210 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2023-11-18T14:11:51.311 INFO:teuthology.orchestra.console.smithi157:Power on completed 2023-11-18T14:11:51.412 INFO:teuthology.provision.fog.smithi157:Waiting for deploy to finish 2023-11-18T14:11:55.200 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:11:55.207 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:11:55.309 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-11-18T14:11:55.314 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2023-11-18T14:11:55.411 INFO:teuthology.orchestra.console.smithi148:Power on completed 2023-11-18T14:11:55.415 INFO:teuthology.orchestra.console.smithi131:Power on completed 2023-11-18T14:11:55.511 INFO:teuthology.provision.fog.smithi148:Waiting for deploy to finish 2023-11-18T14:11:55.516 INFO:teuthology.provision.fog.smithi131:Waiting for deploy to finish 2023-11-18T14:14:26.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-18T14:14:41.227 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-18T14:14:52.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-18T14:14:59.682 WARNING:teuthology.provision.fog.smithi148:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-11-18T14:15:05.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-18T14:15:08.738 WARNING:teuthology.provision.fog.smithi148:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-11-18T14:15:10.690 WARNING:teuthology.provision.fog.smithi157:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-11-18T14:15:14.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-18T14:15:16.694 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-18T14:15:17.826 WARNING:teuthology.provision.fog.smithi148:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-11-18T14:15:19.746 WARNING:teuthology.provision.fog.smithi157:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-11-18T14:15:23.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-18T14:15:25.750 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-18T14:15:26.167 WARNING:teuthology.provision.fog.smithi131:timed out 2023-11-18T14:15:26.914 WARNING:teuthology.provision.fog.smithi148:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-11-18T14:15:28.802 WARNING:teuthology.provision.fog.smithi157:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-11-18T14:15:32.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-18T14:15:32.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-18T14:15:34.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-18T14:15:35.970 WARNING:teuthology.provision.fog.smithi148:[Errno None] Unable to connect to port 22 on 172.21.15.148 2023-11-18T14:15:37.858 WARNING:teuthology.provision.fog.smithi157:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-11-18T14:15:41.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-18T14:15:43.861 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-18T14:15:46.914 WARNING:teuthology.provision.fog.smithi157:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-11-18T14:15:50.626 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131 2023-11-18T14:15:52.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-18T14:15:55.970 WARNING:teuthology.provision.fog.smithi157:[Errno None] Unable to connect to port 22 on 172.21.15.157 2023-11-18T14:15:56.628 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-18T14:15:56.867 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-11-18T14:15:56.867 DEBUG:teuthology.orchestra.run.smithi131:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-11-18T14:16:01.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-18T14:16:01.998 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-18T14:16:02.000 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/provision/fog.py", line 298, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi131 was lost: "while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done" 2023-11-18T14:16:02.002 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-18T14:16:02.133 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ecfb68103460458fbb1e77e0a8db68d3 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/provision/fog.py", line 298, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi131 was lost: "while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done" 2023-11-18T14:16:02.136 INFO:teuthology.nuke:Checking targets against current locks 2023-11-18T14:16:02.194 DEBUG:teuthology.nuke:shortname: smithi131 2023-11-18T14:16:02.195 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-18T14:16:02.219 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-18_13:51:47-smoke-main-distro-default-smithi/7461499', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-18 14:11:45.466589', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKN7BQVhLdrJ0//v6pTk4CDQa3c+vZVNV77GtQkqEJ6z+XwXd1SaLj0kXOpVOVmvaPD0WFPBchdudq/RKhBJoYs='} 2023-11-18T14:16:02.224 DEBUG:teuthology.nuke:shortname: smithi148 2023-11-18T14:16:02.224 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-18T14:16:02.244 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-18_13:51:47-smoke-main-distro-default-smithi/7461499', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-18 14:11:45.468415', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHX3VcRxLDHkum0l6ougXC8C5aLtMC/erwqDckst13sVBr/ZqLGi3s5rnXIfizTEkPJKlh/P52svCEsq/5qSorM='} 2023-11-18T14:16:02.247 DEBUG:teuthology.nuke:shortname: smithi157 2023-11-18T14:16:02.247 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-18T14:16:02.258 INFO:teuthology.orchestra.console.smithi131:Power off 2023-11-18T14:16:02.259 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-18T14:16:02.267 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-18_13:51:47-smoke-main-distro-default-smithi/7461499', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-18 14:11:45.469750', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPrtvqZ7pXF74uRRTT4nv8SqSuRd16ew6CUdoKxMThq4I1c2rS+GvIxA8MN2EJkN6CZPRhsACy/wV+lmhW/Obfk='} 2023-11-18T14:16:02.280 DEBUG:teuthology.orchestra.console.smithi131:power off output: Chassis Power Control: Down/Off 2023-11-18T14:16:02.280 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:16:02.293 INFO:teuthology.orchestra.console.smithi148:Power off 2023-11-18T14:16:02.294 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-18T14:16:02.305 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2023-11-18T14:16:02.315 DEBUG:teuthology.orchestra.console.smithi148:power off output: Chassis Power Control: Down/Off 2023-11-18T14:16:02.315 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:16:02.328 INFO:teuthology.orchestra.console.smithi157:Power off 2023-11-18T14:16:02.328 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-18T14:16:02.336 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-11-18T14:16:02.348 DEBUG:teuthology.orchestra.console.smithi157:power off output: Chassis Power Control: Down/Off 2023-11-18T14:16:02.349 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:16:02.370 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2023-11-18T14:16:06.309 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:16:06.336 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:16:06.371 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:16:06.418 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2023-11-18T14:16:06.444 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-11-18T14:16:06.479 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is off 2023-11-18T14:16:06.580 INFO:teuthology.orchestra.console.smithi157:Power off completed 2023-11-18T14:16:06.730 INFO:teuthology.lock.ops:unlocked: smithi157.front.sepia.ceph.com 2023-11-18T14:16:10.423 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:16:10.445 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:16:10.535 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2023-11-18T14:16:10.555 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is on 2023-11-18T14:16:10.636 INFO:teuthology.orchestra.console.smithi131:Power off completed 2023-11-18T14:16:10.797 INFO:teuthology.lock.ops:unlocked: smithi131.front.sepia.ceph.com 2023-11-18T14:16:14.560 DEBUG:teuthology.orchestra.console.smithi148:pexpect command: ipmitool -H smithi148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-18T14:16:19.459 DEBUG:teuthology.orchestra.console.smithi148:check power output: Chassis Power is off 2023-11-18T14:16:19.560 INFO:teuthology.orchestra.console.smithi148:Power off completed 2023-11-18T14:16:19.713 INFO:teuthology.lock.ops:unlocked: smithi148.front.sepia.ceph.com 2023-11-18T14:16:20.068 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/dispatcher/supervisor.py", line 228, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/provision/fog.py", line 298, in _wait_for_ready self.remote.run(args=cmd, timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi131 was lost: "while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done" 2023-11-18T14:16:20.322 WARNING:teuthology.provision.fog.smithi157:[Errno None] Unable to connect to port 22 on 172.21.15.157