2023-12-07T13:28:17.766 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-07T13:28:17.766 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-07T13:28:17.944 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7feeb82c9dc0>, signals=[15]) 2023-12-07T13:28:17.944 INFO:teuthology.lock.ops:Start node 'smithi037.front.sepia.ceph.com' reimaging 2023-12-07T13:28:17.945 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: reset os type and version on server 2023-12-07T13:28:17.945 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2023-12-07T13:28:17.968 INFO:teuthology.lock.ops:Start node 'smithi140.front.sepia.ceph.com' reimaging 2023-12-07T13:28:17.968 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: reset os type and version on server 2023-12-07T13:28:17.968 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server 2023-12-07T13:28:17.993 INFO:teuthology.lock.ops:Start node 'smithi187.front.sepia.ceph.com' reimaging 2023-12-07T13:28:17.993 INFO:teuthology.lock.ops:Updating [smithi187.front.sepia.ceph.com]: reset os type and version on server 2023-12-07T13:28:17.993 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2023-12-07T13:28:18.086 DEBUG:teuthology.provision.fog.smithi037:Requesting image smithi_rhel_8.6 (ID 5) 2023-12-07T13:28:18.110 DEBUG:teuthology.provision.fog.smithi140:Requesting image smithi_rhel_8.6 (ID 5) 2023-12-07T13:28:18.133 INFO:teuthology.provision.fog.smithi037:Scheduling deploy of rhel 8.6 2023-12-07T13:28:18.148 DEBUG:teuthology.provision.fog.smithi187:Requesting image smithi_rhel_8.6 (ID 5) 2023-12-07T13:28:18.167 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of rhel 8.6 2023-12-07T13:28:18.186 INFO:teuthology.provision.fog.smithi187:Scheduling deploy of rhel 8.6 2023-12-07T13:28:18.338 INFO:teuthology.orchestra.console.smithi037:Power off 2023-12-07T13:28:18.338 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-07T13:28:18.362 DEBUG:teuthology.orchestra.console.smithi037:power off output: Chassis Power Control: Down/Off 2023-12-07T13:28:18.362 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:28:18.384 INFO:teuthology.orchestra.console.smithi187:Power off 2023-12-07T13:28:18.384 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-07T13:28:18.395 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on 2023-12-07T13:28:18.405 INFO:teuthology.orchestra.console.smithi140:Power off 2023-12-07T13:28:18.405 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-07T13:28:18.414 DEBUG:teuthology.orchestra.console.smithi187:power off output: Chassis Power Control: Down/Off 2023-12-07T13:28:18.414 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:28:18.426 DEBUG:teuthology.orchestra.console.smithi140:power off output: Chassis Power Control: Down/Off 2023-12-07T13:28:18.427 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:28:18.440 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2023-12-07T13:28:18.447 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is on 2023-12-07T13:28:22.398 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:28:22.441 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:28:22.449 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:28:22.509 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on 2023-12-07T13:28:22.550 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2023-12-07T13:28:22.557 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is on 2023-12-07T13:28:26.514 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:28:26.551 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:28:26.558 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:28:26.623 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on 2023-12-07T13:28:26.661 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2023-12-07T13:28:26.667 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is off 2023-12-07T13:28:26.768 INFO:teuthology.orchestra.console.smithi140:Power off completed 2023-12-07T13:28:26.869 INFO:teuthology.orchestra.console.smithi140:Power on 2023-12-07T13:28:26.869 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-12-07T13:28:26.889 DEBUG:teuthology.orchestra.console.smithi140:power on output: Chassis Power Control: Up/On 2023-12-07T13:28:26.889 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:28:26.910 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is off 2023-12-07T13:28:30.626 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:28:30.662 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:28:30.737 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2023-12-07T13:28:30.771 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is off 2023-12-07T13:28:30.839 INFO:teuthology.orchestra.console.smithi037:Power off completed 2023-12-07T13:28:30.872 INFO:teuthology.orchestra.console.smithi187:Power off completed 2023-12-07T13:28:30.911 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:28:30.940 INFO:teuthology.orchestra.console.smithi037:Power on 2023-12-07T13:28:30.940 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-12-07T13:28:30.962 DEBUG:teuthology.orchestra.console.smithi037:power on output: Chassis Power Control: Up/On 2023-12-07T13:28:30.963 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:28:30.974 INFO:teuthology.orchestra.console.smithi187:Power on 2023-12-07T13:28:30.974 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-12-07T13:28:30.987 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2023-12-07T13:28:30.995 DEBUG:teuthology.orchestra.console.smithi187:power on output: Chassis Power Control: Up/On 2023-12-07T13:28:30.996 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:28:31.017 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is off 2023-12-07T13:28:31.020 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is off 2023-12-07T13:28:34.990 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:28:35.018 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:28:35.024 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:28:35.098 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on 2023-12-07T13:28:35.127 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2023-12-07T13:28:35.133 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is on 2023-12-07T13:28:35.199 INFO:teuthology.orchestra.console.smithi037:Power on completed 2023-12-07T13:28:35.228 INFO:teuthology.orchestra.console.smithi187:Power on completed 2023-12-07T13:28:35.234 INFO:teuthology.orchestra.console.smithi140:Power on completed 2023-12-07T13:28:35.301 INFO:teuthology.provision.fog.smithi037:Waiting for deploy to finish 2023-12-07T13:28:35.329 INFO:teuthology.provision.fog.smithi187:Waiting for deploy to finish 2023-12-07T13:28:35.334 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish 2023-12-07T13:31:06.362 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:31:09.442 WARNING:teuthology.provision.fog.smithi037:[Errno None] Unable to connect to port 22 on 172.21.15.37 2023-12-07T13:31:15.446 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:31:18.502 WARNING:teuthology.provision.fog.smithi037:[Errno None] Unable to connect to port 22 on 172.21.15.37 2023-12-07T13:31:21.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:31:21.446 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:31:24.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:31:24.514 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:31:24.515 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-07T13:31:27.586 WARNING:teuthology.provision.fog.smithi037:[Errno None] Unable to connect to port 22 on 172.21.15.37 2023-12-07T13:31:30.518 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:31:30.520 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:31:33.571 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-07T13:31:33.571 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:31:33.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:31:36.642 WARNING:teuthology.provision.fog.smithi037:[Errno None] Unable to connect to port 22 on 172.21.15.37 2023-12-07T13:31:39.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:31:39.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:31:42.626 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:31:42.627 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-07T13:31:42.644 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:31:48.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:31:48.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:31:51.682 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:31:51.682 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-07T13:31:57.686 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:31:57.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:32:00.738 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-07T13:32:00.739 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:32:06.742 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:32:06.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:32:09.794 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-07T13:32:09.795 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:32:15.801 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:32:15.804 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:32:18.882 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:32:24.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:32:27.938 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:32:33.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:32:34.274 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-07T13:32:40.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:32:42.649 WARNING:teuthology.provision.fog.smithi037:timed out 2023-12-07T13:32:43.330 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-07T13:32:48.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:32:48.997 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-12-07T13:32:48.997 DEBUG:teuthology.orchestra.run.smithi037:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-12-07T13:32:49.287 INFO:teuthology.provision.fog.smithi037:Node is ready 2023-12-07T13:32:49.287 DEBUG:teuthology.orchestra.run.smithi037:> hostname 2023-12-07T13:32:49.306 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037.front.sepia.ceph.com 2023-12-07T13:32:49.307 DEBUG:teuthology.orchestra.run.smithi037:> grep smithi037.front.sepia.ceph.com /etc/hosts 2023-12-07T13:32:49.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:32:49.365 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37 smithi037.front.sepia.ceph.com smithi037 2023-12-07T13:32:49.366 DEBUG:teuthology.orchestra.run.smithi037:> sudo hostname smithi037 2023-12-07T13:32:49.482 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i -e 's/smithi037.front.sepia.ceph.com/smithi037/g' /etc/hosts 2023-12-07T13:32:49.512 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i -e 's/172.21.15.37/172.21.15.37/g' /etc/hosts 2023-12-07T13:32:49.663 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i -e 's/smithi037.front.sepia.ceph.com/smithi037/g' /etc/hostname 2023-12-07T13:32:49.691 DEBUG:teuthology.orchestra.run.smithi037:> sudo hostname smithi037 2023-12-07T13:32:49.762 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release 2023-12-07T13:32:49.820 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="Red Hat Enterprise Linux" 2023-12-07T13:32:49.821 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="8.6 (Ootpa)" 2023-12-07T13:32:49.821 INFO:teuthology.orchestra.run.smithi037.stdout:ID="rhel" 2023-12-07T13:32:49.821 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="fedora" 2023-12-07T13:32:49.821 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="8.6" 2023-12-07T13:32:49.821 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el8" 2023-12-07T13:32:49.821 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-12-07T13:32:49.821 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31" 2023-12-07T13:32:49.821 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-12-07T13:32:49.821 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://www.redhat.com/" 2023-12-07T13:32:49.821 INFO:teuthology.orchestra.run.smithi037.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-12-07T13:32:49.822 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-07T13:32:49.822 INFO:teuthology.orchestra.run.smithi037.stdout: 2023-12-07T13:32:49.822 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-07T13:32:49.822 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-12-07T13:32:49.822 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-12-07T13:32:49.823 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-12-07T13:32:49.823 INFO:teuthology.provision.fog.smithi037:Deploy complete! 2023-12-07T13:32:49.825 DEBUG:teuthology.parallel:result is None 2023-12-07T13:32:52.390 WARNING:teuthology.provision.fog.smithi187:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-07T13:32:58.394 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:32:58.737 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-12-07T13:32:59.122 DEBUG:teuthology.orchestra.run.smithi187:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-12-07T13:32:59.176 INFO:teuthology.provision.fog.smithi187:Node is ready 2023-12-07T13:32:59.176 DEBUG:teuthology.orchestra.run.smithi187:> hostname 2023-12-07T13:32:59.192 INFO:teuthology.orchestra.run.smithi187.stdout:smithi187.front.sepia.ceph.com 2023-12-07T13:32:59.193 DEBUG:teuthology.orchestra.run.smithi187:> grep smithi187.front.sepia.ceph.com /etc/hosts 2023-12-07T13:32:59.249 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.15.187 smithi187.front.sepia.ceph.com smithi187 2023-12-07T13:32:59.249 DEBUG:teuthology.orchestra.run.smithi187:> sudo hostname smithi187 2023-12-07T13:32:59.390 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i -e 's/smithi187.front.sepia.ceph.com/smithi187/g' /etc/hosts 2023-12-07T13:32:59.417 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i -e 's/172.21.15.187/172.21.15.187/g' /etc/hosts 2023-12-07T13:32:59.484 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i -e 's/smithi187.front.sepia.ceph.com/smithi187/g' /etc/hostname 2023-12-07T13:32:59.550 DEBUG:teuthology.orchestra.run.smithi187:> sudo hostname smithi187 2023-12-07T13:32:59.616 DEBUG:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2023-12-07T13:32:59.671 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="Red Hat Enterprise Linux" 2023-12-07T13:32:59.671 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="8.6 (Ootpa)" 2023-12-07T13:32:59.672 INFO:teuthology.orchestra.run.smithi187.stdout:ID="rhel" 2023-12-07T13:32:59.672 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="fedora" 2023-12-07T13:32:59.672 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="8.6" 2023-12-07T13:32:59.672 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el8" 2023-12-07T13:32:59.672 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-12-07T13:32:59.672 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2023-12-07T13:32:59.672 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-12-07T13:32:59.672 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://www.redhat.com/" 2023-12-07T13:32:59.672 INFO:teuthology.orchestra.run.smithi187.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-12-07T13:32:59.672 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-07T13:32:59.672 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-07T13:32:59.672 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-07T13:32:59.672 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-12-07T13:32:59.673 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-12-07T13:32:59.673 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-12-07T13:32:59.673 INFO:teuthology.provision.fog.smithi187:Deploy complete! 2023-12-07T13:32:59.673 DEBUG:teuthology.parallel:result is None 2023-12-07T13:33:33.946 WARNING:teuthology.provision.fog.smithi140:timed out 2023-12-07T13:33:39.948 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:33:43.014 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:33:49.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:33:52.070 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:33:58.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:34:01.158 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:34:07.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:34:10.249 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:34:16.252 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:34:19.334 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:34:25.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:34:28.418 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:34:34.420 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:34:37.474 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:34:43.476 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:34:46.530 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:34:52.532 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:34:55.590 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:35:01.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:35:04.678 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:35:10.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:35:13.730 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:35:19.732 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:35:22.786 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:35:28.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:35:31.845 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:35:37.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:35:37.915 ERROR:paramiko.transport:Unknown exception: The socket has already been closed by another greenlet 2023-12-07T13:35:37.917 ERROR:paramiko.transport:Traceback (most recent call last): 2023-12-07T13:35:37.917 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run 2023-12-07T13:35:37.917 ERROR:paramiko.transport: ptype, m = self.packetizer.read_message() 2023-12-07T13:35:37.917 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message 2023-12-07T13:35:37.917 ERROR:paramiko.transport: header = self.read_all(self.__block_size_in, check_rekey=True) 2023-12-07T13:35:37.917 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all 2023-12-07T13:35:37.918 ERROR:paramiko.transport: x = self.__socket.recv(n) 2023-12-07T13:35:37.918 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 663, in recv 2023-12-07T13:35:37.918 ERROR:paramiko.transport: self._wait(self._read_event) 2023-12-07T13:35:37.918 ERROR:paramiko.transport: File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket 2023-12-07T13:35:37.918 ERROR:paramiko.transport: File "src/gevent/_hub_primitives.py", line 321, in gevent._gevent_c_hub_primitives.wait_on_socket 2023-12-07T13:35:37.918 ERROR:paramiko.transport:gevent.exceptions.ConcurrentObjectUseError: The socket has already been closed by another greenlet 2023-12-07T13:35:37.918 ERROR:paramiko.transport: 2023-12-07T13:35:40.898 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:35:46.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:35:49.954 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:35:55.958 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:35:59.010 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:36:05.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:36:08.066 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:36:14.070 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:36:17.126 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:36:23.133 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:36:26.210 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:36:32.214 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:36:35.270 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:36:41.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:36:44.354 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:36:50.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:36:53.414 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:36:59.418 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:37:02.498 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:37:08.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:37:11.554 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:37:17.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:37:20.611 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:37:26.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:37:29.666 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:37:35.672 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:37:38.754 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:37:44.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:37:47.810 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:37:53.814 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:37:56.866 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:38:02.870 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:38:05.922 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:38:11.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:38:14.978 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:38:20.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:38:24.038 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:38:30.042 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:38:33.126 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:38:39.132 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:38:42.214 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:38:48.220 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:38:51.298 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:38:57.305 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:39:00.386 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:39:06.393 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:39:09.478 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:39:15.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:39:18.566 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:39:24.570 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:39:27.650 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:39:33.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:39:36.710 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:39:42.714 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:39:45.798 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:39:51.804 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:39:54.886 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:40:00.890 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:40:03.974 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:40:09.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:40:13.058 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:40:19.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:40:22.114 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:40:28.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:40:31.174 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:40:37.178 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:40:40.258 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:40:46.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:40:49.314 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:40:55.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:40:58.370 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:41:04.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:41:07.426 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:41:13.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:41:16.486 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:41:22.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:41:25.574 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:41:31.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:41:34.662 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:41:40.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:41:43.746 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:41:49.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:41:52.834 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:41:58.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:42:01.926 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:42:07.930 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:42:11.013 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:42:17.020 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:42:20.098 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:42:26.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:42:29.158 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:42:35.162 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:42:38.242 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:42:44.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:42:47.330 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:42:53.337 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:42:56.418 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:43:02.423 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:43:05.474 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:43:11.478 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:43:14.530 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:43:20.534 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:43:23.618 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:43:29.622 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:43:32.674 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:43:38.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:43:41.733 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:43:47.742 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:43:50.822 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:43:56.826 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:43:59.906 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:44:05.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:44:08.962 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:44:14.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:44:18.021 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:44:24.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:44:27.106 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:44:33.111 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:44:36.162 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:44:42.170 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:44:45.254 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:44:51.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:44:54.338 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:45:00.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:45:03.394 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:45:09.398 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:45:12.450 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:45:18.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:45:21.538 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:45:27.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:45:30.626 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:45:36.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:45:39.686 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:45:45.690 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:45:48.770 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:45:54.774 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:45:57.826 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:46:03.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:46:06.882 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:46:12.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:46:15.970 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:46:21.974 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:46:25.026 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:46:31.030 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:46:34.082 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:46:40.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:46:43.138 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:46:49.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:46:52.194 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:46:58.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:47:01.254 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:47:07.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:47:10.338 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:47:16.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T13:47:19.394 WARNING:teuthology.provision.fog.smithi140:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-07T13:47:19.396 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/provision/fog.py", line 275, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2023-12-07T13:47:19.397 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-07T13:47:19.546 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=477aabd74ef84550813801073cc718ae Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/provision/fog.py", line 275, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2023-12-07T13:47:19.548 INFO:teuthology.nuke:Checking targets against current locks 2023-12-07T13:47:19.604 DEBUG:teuthology.nuke:shortname: smithi037 2023-12-07T13:47:19.604 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-07T13:47:19.630 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-12-06_15:12:46-fs-wip-vshankar-testing-20231206.125818-testing-default-smithi/7480487', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-07 13:28:16.856246', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMdIoyaM4KS8LML2MQWzmbIhJ+mGh1kbz1bmHKrE2NaaRJW0nu82qqSctBSDllQgCJuS1acyOov3M/mNQUlvAc8='} 2023-12-07T13:47:19.634 DEBUG:teuthology.nuke:shortname: smithi140 2023-12-07T13:47:19.634 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-07T13:47:19.656 DEBUG:teuthology.nuke:shortname: smithi187 2023-12-07T13:47:19.657 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-07T13:47:19.661 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi140.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-12-06_15:12:46-fs-wip-vshankar-testing-20231206.125818-testing-default-smithi/7480487', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-07 13:28:16.857807', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG7u/WBh+/hEo18mZ/MhRtQjpMs0FrHFKkc8MHnK7PJ4LlEogd+0TWzCiNZu0swzFhqJgg0UmWr4YEjDItqbof8='} 2023-12-07T13:47:19.693 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-12-06_15:12:46-fs-wip-vshankar-testing-20231206.125818-testing-default-smithi/7480487', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-07 13:28:16.859277', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBISpA6QTL6r9BqopSbvLUXNrozTruvNJbU4JCuOeJvIApKH1p1ux5nI67Vj16bsAc9FdKOKVVxOQhoSdUtHeEzc='} 2023-12-07T13:47:19.699 INFO:teuthology.orchestra.console.smithi037:Power off 2023-12-07T13:47:19.700 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-07T13:47:19.726 DEBUG:teuthology.orchestra.console.smithi037:power off output: Chassis Power Control: Down/Off 2023-12-07T13:47:19.726 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:47:19.743 INFO:teuthology.orchestra.console.smithi140:Power off 2023-12-07T13:47:19.743 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-07T13:47:19.755 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on 2023-12-07T13:47:19.765 DEBUG:teuthology.orchestra.console.smithi140:power off output: Chassis Power Control: Down/Off 2023-12-07T13:47:19.765 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:47:19.783 INFO:teuthology.orchestra.console.smithi187:Power off 2023-12-07T13:47:19.784 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-07T13:47:19.793 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is on 2023-12-07T13:47:19.810 DEBUG:teuthology.orchestra.console.smithi187:power off output: Chassis Power Control: Down/Off 2023-12-07T13:47:19.810 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:47:19.833 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2023-12-07T13:47:23.758 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:47:23.794 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:47:23.834 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:47:23.868 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on 2023-12-07T13:47:23.904 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is on 2023-12-07T13:47:23.943 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2023-12-07T13:47:27.870 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:47:27.905 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:47:27.945 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:47:27.980 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on 2023-12-07T13:47:28.015 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is on 2023-12-07T13:47:28.054 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is off 2023-12-07T13:47:28.155 INFO:teuthology.orchestra.console.smithi187:Power off completed 2023-12-07T13:47:28.313 INFO:teuthology.lock.ops:unlocked: smithi187.front.sepia.ceph.com 2023-12-07T13:47:31.982 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:47:32.017 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T13:47:32.092 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2023-12-07T13:47:32.127 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is off 2023-12-07T13:47:32.193 INFO:teuthology.orchestra.console.smithi037:Power off completed 2023-12-07T13:47:32.228 INFO:teuthology.orchestra.console.smithi140:Power off completed 2023-12-07T13:47:32.344 INFO:teuthology.lock.ops:unlocked: smithi037.front.sepia.ceph.com 2023-12-07T13:47:32.378 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com 2023-12-07T13:47:32.737 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/dispatcher/supervisor.py", line 58, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/dispatcher/supervisor.py", line 232, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/provision/fog.py", line 90, in create self._wait_for_ready() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/provision/fog.py", line 275, in _wait_for_ready while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds