2024-03-09T11:56:14.129 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-09T11:56:14.130 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-09T11:56:14.278 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8539731c10>, signals=[15]) 2024-03-09T11:56:14.279 INFO:teuthology.lock.ops:Start node 'smithi158.front.sepia.ceph.com' reimaging 2024-03-09T11:56:14.279 INFO:teuthology.lock.ops:Updating [smithi158.front.sepia.ceph.com]: reset os type and version on server 2024-03-09T11:56:14.279 INFO:teuthology.lock.ops:Updating smithi158.front.sepia.ceph.com on lock server 2024-03-09T11:56:14.313 INFO:teuthology.lock.ops:Start node 'smithi177.front.sepia.ceph.com' reimaging 2024-03-09T11:56:14.314 INFO:teuthology.lock.ops:Updating [smithi177.front.sepia.ceph.com]: reset os type and version on server 2024-03-09T11:56:14.314 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-03-09T11:56:14.430 DEBUG:teuthology.provision.fog.smithi158:Requesting image smithi_centos_9.stream (ID 10) 2024-03-09T11:56:14.449 DEBUG:teuthology.provision.fog.smithi177:Requesting image smithi_centos_9.stream (ID 10) 2024-03-09T11:56:14.476 INFO:teuthology.provision.fog.smithi158:Scheduling deploy of centos 9.stream 2024-03-09T11:56:14.512 INFO:teuthology.provision.fog.smithi177:Scheduling deploy of centos 9.stream 2024-03-09T11:56:14.705 INFO:teuthology.orchestra.console.smithi158:Power off 2024-03-09T11:56:14.705 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-09T11:56:14.726 DEBUG:teuthology.orchestra.console.smithi158:power off output: Chassis Power Control: Down/Off 2024-03-09T11:56:14.727 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T11:56:14.747 INFO:teuthology.orchestra.console.smithi177:Power off 2024-03-09T11:56:14.748 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-09T11:56:14.755 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is off 2024-03-09T11:56:14.770 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2024-03-09T11:56:14.770 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T11:56:14.792 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-09T11:56:14.857 INFO:teuthology.orchestra.console.smithi158:Power off completed 2024-03-09T11:56:14.958 INFO:teuthology.orchestra.console.smithi158:Power on 2024-03-09T11:56:14.958 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-09T11:56:14.979 DEBUG:teuthology.orchestra.console.smithi158:power on output: Chassis Power Control: Up/On 2024-03-09T11:56:14.979 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T11:56:15.000 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is off 2024-03-09T11:56:18.796 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T11:56:18.906 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-09T11:56:19.001 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T11:56:19.110 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is on 2024-03-09T11:56:19.210 INFO:teuthology.orchestra.console.smithi158:Power on completed 2024-03-09T11:56:19.311 INFO:teuthology.provision.fog.smithi158:Waiting for deploy to finish 2024-03-09T11:56:22.910 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T11:56:23.019 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2024-03-09T11:56:23.121 INFO:teuthology.orchestra.console.smithi177:Power off completed 2024-03-09T11:56:23.222 INFO:teuthology.orchestra.console.smithi177:Power on 2024-03-09T11:56:23.222 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-09T11:56:23.246 DEBUG:teuthology.orchestra.console.smithi177:power on output: Chassis Power Control: Up/On 2024-03-09T11:56:23.247 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T11:56:23.271 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2024-03-09T11:56:27.276 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T11:56:27.385 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2024-03-09T11:56:31.390 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T11:56:31.498 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-09T11:56:31.599 INFO:teuthology.orchestra.console.smithi177:Power on completed 2024-03-09T11:56:31.700 INFO:teuthology.provision.fog.smithi177:Waiting for deploy to finish 2024-03-09T11:59:02.565 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T11:59:05.630 WARNING:teuthology.provision.fog.smithi177:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-09T11:59:11.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T11:59:14.686 WARNING:teuthology.provision.fog.smithi177:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-09T11:59:20.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T11:59:23.742 WARNING:teuthology.provision.fog.smithi177:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-09T11:59:29.747 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T11:59:32.802 WARNING:teuthology.provision.fog.smithi177:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-09T11:59:38.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T11:59:41.886 WARNING:teuthology.provision.fog.smithi177:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-09T11:59:47.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T11:59:50.942 WARNING:teuthology.provision.fog.smithi177:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-09T11:59:56.949 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:00:15.394 WARNING:teuthology.provision.fog.smithi177:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-09T12:00:21.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:00:24.478 WARNING:teuthology.provision.fog.smithi177:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-09T12:00:30.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:00:33.534 WARNING:teuthology.provision.fog.smithi177:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-09T12:00:39.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:00:39.867 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-09T12:00:39.868 DEBUG:teuthology.orchestra.run.smithi177:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-09T12:00:55.175 INFO:teuthology.provision.fog.smithi177:Node is ready 2024-03-09T12:00:55.176 DEBUG:teuthology.orchestra.run.smithi177:> hostname 2024-03-09T12:00:55.193 INFO:teuthology.orchestra.run.smithi177.stdout:smithi046 2024-03-09T12:00:55.194 DEBUG:teuthology.orchestra.run.smithi177:> grep smithi046 /etc/hosts 2024-03-09T12:00:55.251 INFO:teuthology.orchestra.run.smithi177.stdout:172.21.15.46 smithi046.front.sepia.ceph.com smithi046 2024-03-09T12:00:55.251 DEBUG:teuthology.orchestra.run.smithi177:> sudo hostname smithi177 2024-03-09T12:00:55.396 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i -e 's/smithi046/smithi177/g' /etc/hosts 2024-03-09T12:00:55.427 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i -e 's/172.21.15.46/172.21.15.177/g' /etc/hosts 2024-03-09T12:00:55.497 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i -e 's/smithi046/smithi177/g' /etc/hostname 2024-03-09T12:00:55.565 DEBUG:teuthology.orchestra.run.smithi177:> sudo hostname smithi177 2024-03-09T12:00:55.633 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-03-09T12:00:55.688 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-03-09T12:00:55.688 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2024-03-09T12:00:55.688 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-03-09T12:00:55.688 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-03-09T12:00:55.688 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2024-03-09T12:00:55.688 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2024-03-09T12:00:55.688 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-09T12:00:55.688 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-03-09T12:00:55.688 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2024-03-09T12:00:55.688 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-09T12:00:55.688 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-03-09T12:00:55.688 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-09T12:00:55.688 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-09T12:00:55.688 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-09T12:00:55.689 INFO:teuthology.provision.fog.smithi177:Deploy complete! 2024-03-09T12:00:55.691 DEBUG:teuthology.parallel:result is None 2024-03-09T12:25:43.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:25:43.419 ERROR:teuthology.orchestra.connection:Error authenticating with smithi158.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-03-09T12:25:47.550 ERROR:teuthology.orchestra.connection:Error authenticating with smithi158.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-03-09T12:26:54.554 WARNING:teuthology.provision.fog.smithi158:timed out 2024-03-09T12:27:00.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:27:03.618 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:27:09.620 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:27:12.670 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:27:18.672 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:27:21.730 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:27:27.732 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:27:30.782 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:27:36.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:28:36.786 WARNING:teuthology.provision.fog.smithi158:timed out 2024-03-09T12:28:42.788 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:29:01.342 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:29:07.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:29:10.398 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:29:16.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:29:19.454 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:29:25.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:29:28.510 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:29:34.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:29:37.566 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:29:43.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:29:46.622 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:29:52.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:29:55.678 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:30:01.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:30:04.734 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:30:10.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:30:13.790 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:30:19.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:30:22.850 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:30:28.852 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:30:31.902 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:30:37.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:30:40.958 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:30:46.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:30:50.014 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:30:56.016 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:30:59.070 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:31:05.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:31:08.130 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:31:14.132 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:31:17.214 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:31:23.216 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:31:26.270 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:31:32.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:32:32.274 WARNING:teuthology.provision.fog.smithi158:timed out 2024-03-09T12:32:38.276 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:32:56.610 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:33:02.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:34:02.615 WARNING:teuthology.provision.fog.smithi158:timed out 2024-03-09T12:34:08.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:35:08.620 WARNING:teuthology.provision.fog.smithi158:timed out 2024-03-09T12:35:14.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:36:14.624 WARNING:teuthology.provision.fog.smithi158:timed out 2024-03-09T12:36:20.625 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:37:20.628 WARNING:teuthology.provision.fog.smithi158:timed out 2024-03-09T12:37:26.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:38:26.634 WARNING:teuthology.provision.fog.smithi158:timed out 2024-03-09T12:38:32.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:38:51.166 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:38:57.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:39:00.222 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:39:06.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:39:09.278 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:39:15.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:39:18.338 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:39:24.340 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:39:27.390 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:39:33.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:39:36.450 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:39:42.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:39:45.506 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:39:51.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:39:54.561 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:40:00.562 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:40:03.614 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:40:09.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:40:12.670 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:40:18.672 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:40:21.726 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:40:27.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:40:30.783 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:40:36.785 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:40:39.843 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:40:45.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:40:48.894 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:40:54.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:40:57.954 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:41:03.956 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:41:07.006 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:41:13.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:41:16.062 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:41:22.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:41:25.118 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:41:31.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:41:34.174 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:41:40.176 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:41:43.234 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:41:49.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:41:52.286 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:41:58.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:42:01.342 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:42:07.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:42:10.402 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:42:16.404 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:42:19.454 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:42:25.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:42:28.514 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:42:34.516 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:42:37.566 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:42:43.567 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:42:46.622 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:42:52.623 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:42:55.678 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:43:01.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:43:04.734 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:43:10.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:43:13.790 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:43:19.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:43:22.850 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:43:28.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:43:31.902 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:43:37.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:43:40.958 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:43:46.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:43:50.014 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:43:56.016 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:43:59.070 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:44:05.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:44:08.126 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:44:14.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:44:17.182 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:44:23.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:44:26.238 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:44:32.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:44:35.294 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:44:41.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:44:44.350 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:44:50.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:44:53.406 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:44:59.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:45:02.462 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:45:08.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:46:08.467 WARNING:teuthology.provision.fog.smithi158:timed out 2024-03-09T12:46:14.469 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:47:14.475 WARNING:teuthology.provision.fog.smithi158:timed out 2024-03-09T12:47:20.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:48:20.480 WARNING:teuthology.provision.fog.smithi158:timed out 2024-03-09T12:48:26.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:49:26.485 WARNING:teuthology.provision.fog.smithi158:timed out 2024-03-09T12:49:32.488 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:50:32.491 WARNING:teuthology.provision.fog.smithi158:timed out 2024-03-09T12:50:38.493 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:51:38.496 WARNING:teuthology.provision.fog.smithi158:timed out 2024-03-09T12:51:44.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:52:44.501 WARNING:teuthology.provision.fog.smithi158:timed out 2024-03-09T12:52:50.504 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:53:09.022 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:53:15.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:53:18.078 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:53:24.080 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:53:27.134 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:53:33.135 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:53:36.190 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:53:42.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:53:45.246 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:53:51.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:53:54.305 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:54:00.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:54:03.358 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:54:09.360 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:54:12.418 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:54:18.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:54:21.474 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:54:27.476 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:54:30.526 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:54:36.528 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:54:39.586 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:54:45.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:54:48.642 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:54:54.644 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:54:57.694 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:55:03.695 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:55:06.750 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:55:12.752 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:55:15.806 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:55:21.808 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:55:24.862 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:55:30.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:55:33.922 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:55:39.924 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:55:42.974 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:55:48.976 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:55:52.034 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:55:58.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:56:01.090 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:56:07.092 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:56:10.142 WARNING:teuthology.provision.fog.smithi158:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-03-09T12:56:10.144 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 275, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2024-03-09T12:56:10.146 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-09T12:56:10.427 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a4bb4beb45174852aa3501ccad73c563 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 275, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2024-03-09T12:56:10.429 INFO:teuthology.nuke:Checking targets against current locks 2024-03-09T12:56:10.490 DEBUG:teuthology.nuke:shortname: smithi158 2024-03-09T12:56:10.491 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-09T12:56:10.506 DEBUG:teuthology.nuke:shortname: smithi177 2024-03-09T12:56:10.506 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-09T12:56:10.513 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi158.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-08_16:17:06-fs-wip-yuri10-testing-2024-03-07-1242-reef-distro-default-smithi/7586977', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-09 11:56:13.065868', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLoPIZzhPzFDz62h4ys3KqHuE5tcegHyuQ2BhaPwdOGWuqg+jSG/lYfp4qC/ZbnwUtelFzwIK4Gyx5Rssvxzkx8='} 2024-03-09T12:56:10.529 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-08_16:17:06-fs-wip-yuri10-testing-2024-03-07-1242-reef-distro-default-smithi/7586977', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-09 11:56:13.067577', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPt6WGbq0ItVZCJJ6hvAjAFhGeD3GSKAWC+ckYZlZdxfPwxBJEtlPDGExtyVELBvZgYYW4iyh/7TB50+lWYjaKI='} 2024-03-09T12:56:10.559 INFO:teuthology.orchestra.console.smithi158:Power off 2024-03-09T12:56:10.560 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-09T12:56:10.579 INFO:teuthology.orchestra.console.smithi177:Power off 2024-03-09T12:56:10.579 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-09T12:56:10.589 DEBUG:teuthology.orchestra.console.smithi158:power off output: Chassis Power Control: Down/Off 2024-03-09T12:56:10.589 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T12:56:10.602 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2024-03-09T12:56:10.603 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T12:56:10.612 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is on 2024-03-09T12:56:10.626 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-09T12:56:14.613 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T12:56:14.627 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T12:56:14.723 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is off 2024-03-09T12:56:14.737 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-09T12:56:14.824 INFO:teuthology.orchestra.console.smithi158:Power off completed 2024-03-09T12:56:14.979 INFO:teuthology.lock.ops:unlocked: smithi158.front.sepia.ceph.com 2024-03-09T12:56:18.738 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T12:56:18.848 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-09T12:56:22.849 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T12:56:22.962 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2024-03-09T12:56:23.063 INFO:teuthology.orchestra.console.smithi177:Power off completed 2024-03-09T12:56:23.220 INFO:teuthology.lock.ops:unlocked: smithi177.front.sepia.ceph.com 2024-03-09T12:56:23.505 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/provision/fog.py", line 275, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds