2024-01-08T06:53:52.436 INFO:root:teuthology version: 0.0.1.dev228+g01dd171 2024-01-08T06:53:52.436 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-08T06:53:52.558 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0619674940>, signals=[15]) 2024-01-08T06:53:52.558 INFO:teuthology.lock.ops:Start node 'smithi026.front.sepia.ceph.com' reimaging 2024-01-08T06:53:52.558 INFO:teuthology.lock.ops:Updating [smithi026.front.sepia.ceph.com]: reset os type and version on server 2024-01-08T06:53:52.558 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2024-01-08T06:53:52.582 INFO:teuthology.lock.ops:Start node 'smithi176.front.sepia.ceph.com' reimaging 2024-01-08T06:53:52.582 INFO:teuthology.lock.ops:Updating [smithi176.front.sepia.ceph.com]: reset os type and version on server 2024-01-08T06:53:52.582 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-01-08T06:53:52.687 DEBUG:teuthology.provision.fog.smithi026:Requesting image smithi_rhel_8.6 (ID 5) 2024-01-08T06:53:52.709 DEBUG:teuthology.provision.fog.smithi176:Requesting image smithi_rhel_8.6 (ID 5) 2024-01-08T06:53:52.733 INFO:teuthology.provision.fog.smithi026:Scheduling deploy of rhel 8.6 2024-01-08T06:53:52.764 INFO:teuthology.provision.fog.smithi176:Scheduling deploy of rhel 8.6 2024-01-08T06:53:53.442 INFO:teuthology.orchestra.console.smithi026:Power off 2024-01-08T06:53:53.443 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-08T06:53:53.455 INFO:teuthology.orchestra.console.smithi176:Power off 2024-01-08T06:53:53.456 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-08T06:53:53.466 DEBUG:teuthology.orchestra.console.smithi026:power off output: Chassis Power Control: Down/Off 2024-01-08T06:53:53.466 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-08T06:53:53.476 DEBUG:teuthology.orchestra.console.smithi176:power off output: Chassis Power Control: Down/Off 2024-01-08T06:53:53.476 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-08T06:53:53.487 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-01-08T06:53:53.512 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-01-08T06:53:57.492 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-08T06:53:57.513 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-08T06:53:57.601 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-01-08T06:53:57.621 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-01-08T06:54:01.607 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-08T06:54:01.622 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-08T06:54:01.715 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off 2024-01-08T06:54:01.729 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-01-08T06:54:01.817 INFO:teuthology.orchestra.console.smithi026:Power off completed 2024-01-08T06:54:01.917 INFO:teuthology.orchestra.console.smithi026:Power on 2024-01-08T06:54:01.918 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-01-08T06:54:01.940 DEBUG:teuthology.orchestra.console.smithi026:power on output: Chassis Power Control: Up/On 2024-01-08T06:54:01.940 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-08T06:54:01.959 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off 2024-01-08T06:54:05.734 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-08T06:54:05.846 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2024-01-08T06:54:05.947 INFO:teuthology.orchestra.console.smithi176:Power off completed 2024-01-08T06:54:05.960 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-08T06:54:06.048 INFO:teuthology.orchestra.console.smithi176:Power on 2024-01-08T06:54:06.049 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-01-08T06:54:06.067 DEBUG:teuthology.orchestra.console.smithi176:power on output: Chassis Power Control: Up/On 2024-01-08T06:54:06.067 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-08T06:54:06.074 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off 2024-01-08T06:54:06.087 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2024-01-08T06:54:10.080 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-08T06:54:10.088 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-08T06:54:10.189 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-01-08T06:54:10.196 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2024-01-08T06:54:10.289 INFO:teuthology.orchestra.console.smithi026:Power on completed 2024-01-08T06:54:10.390 INFO:teuthology.provision.fog.smithi026:Waiting for deploy to finish 2024-01-08T06:54:14.201 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-08T06:54:14.311 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-01-08T06:54:14.412 INFO:teuthology.orchestra.console.smithi176:Power on completed 2024-01-08T06:54:14.513 INFO:teuthology.provision.fog.smithi176:Waiting for deploy to finish 2024-01-08T07:00:27.973 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:00:31.036 WARNING:teuthology.provision.fog.smithi176:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-01-08T07:00:32.715 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:00:35.773 WARNING:teuthology.provision.fog.smithi026:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-01-08T07:00:37.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:00:40.096 WARNING:teuthology.provision.fog.smithi176:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-01-08T07:00:41.776 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:00:44.832 WARNING:teuthology.provision.fog.smithi026:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-01-08T07:00:46.099 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:00:49.184 WARNING:teuthology.provision.fog.smithi176:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-01-08T07:00:50.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:00:53.920 WARNING:teuthology.provision.fog.smithi026:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-01-08T07:00:55.187 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:00:58.268 WARNING:teuthology.provision.fog.smithi176:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-01-08T07:00:59.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:01:03.008 WARNING:teuthology.provision.fog.smithi026:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-01-08T07:01:04.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:01:07.324 WARNING:teuthology.provision.fog.smithi176:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-01-08T07:01:09.011 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:01:12.092 WARNING:teuthology.provision.fog.smithi026:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-01-08T07:01:13.326 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:01:16.380 WARNING:teuthology.provision.fog.smithi176:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-01-08T07:01:18.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:01:22.384 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:01:25.436 WARNING:teuthology.provision.fog.smithi176:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-01-08T07:01:31.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:01:49.757 WARNING:teuthology.provision.fog.smithi176:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-01-08T07:01:55.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:01:58.812 WARNING:teuthology.provision.fog.smithi176:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-01-08T07:02:04.819 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:02:07.900 WARNING:teuthology.provision.fog.smithi176:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-01-08T07:02:13.905 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:02:14.321 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-01-08T07:02:14.321 DEBUG:teuthology.orchestra.run.smithi176:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-01-08T07:02:18.098 WARNING:teuthology.provision.fog.smithi026:timed out 2024-01-08T07:02:24.100 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:02:24.354 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-01-08T07:02:24.355 DEBUG:teuthology.orchestra.run.smithi026:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-01-08T07:02:24.632 INFO:teuthology.provision.fog.smithi026:Node is ready 2024-01-08T07:02:24.633 DEBUG:teuthology.orchestra.run.smithi026:> hostname 2024-01-08T07:02:24.654 INFO:teuthology.orchestra.run.smithi026.stdout:smithi026.front.sepia.ceph.com 2024-01-08T07:02:24.654 DEBUG:teuthology.orchestra.run.smithi026:> grep smithi026.front.sepia.ceph.com /etc/hosts 2024-01-08T07:02:24.715 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26 smithi026.front.sepia.ceph.com smithi026 2024-01-08T07:02:24.716 DEBUG:teuthology.orchestra.run.smithi026:> sudo hostname smithi026 2024-01-08T07:02:24.842 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i -e 's/smithi026.front.sepia.ceph.com/smithi026/g' /etc/hosts 2024-01-08T07:02:24.876 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i -e 's/172.21.15.26/172.21.15.26/g' /etc/hosts 2024-01-08T07:02:24.951 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i -e 's/smithi026.front.sepia.ceph.com/smithi026/g' /etc/hostname 2024-01-08T07:02:25.026 DEBUG:teuthology.orchestra.run.smithi026:> sudo hostname smithi026 2024-01-08T07:02:25.100 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2024-01-08T07:02:25.162 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="Red Hat Enterprise Linux" 2024-01-08T07:02:25.162 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="8.6 (Ootpa)" 2024-01-08T07:02:25.162 INFO:teuthology.orchestra.run.smithi026.stdout:ID="rhel" 2024-01-08T07:02:25.162 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE="fedora" 2024-01-08T07:02:25.162 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="8.6" 2024-01-08T07:02:25.162 INFO:teuthology.orchestra.run.smithi026.stdout:PLATFORM_ID="platform:el8" 2024-01-08T07:02:25.162 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-08T07:02:25.162 INFO:teuthology.orchestra.run.smithi026.stdout:ANSI_COLOR="0;31" 2024-01-08T07:02:25.162 INFO:teuthology.orchestra.run.smithi026.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-08T07:02:25.162 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://www.redhat.com/" 2024-01-08T07:02:25.163 INFO:teuthology.orchestra.run.smithi026.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-08T07:02:25.163 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-08T07:02:25.163 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-01-08T07:02:25.163 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-08T07:02:25.163 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-08T07:02:25.163 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-08T07:02:25.163 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-08T07:02:25.164 INFO:teuthology.provision.fog.smithi026:Deploy complete! 2024-01-08T07:02:25.165 DEBUG:teuthology.parallel:result is None 2024-01-08T07:02:30.034 INFO:teuthology.provision.fog.smithi176:Node is ready 2024-01-08T07:02:30.034 DEBUG:teuthology.orchestra.run.smithi176:> hostname 2024-01-08T07:02:30.056 INFO:teuthology.orchestra.run.smithi176.stdout:smithi176.front.sepia.ceph.com 2024-01-08T07:02:30.056 DEBUG:teuthology.orchestra.run.smithi176:> grep smithi176.front.sepia.ceph.com /etc/hosts 2024-01-08T07:02:30.116 INFO:teuthology.orchestra.run.smithi176.stdout:172.21.15.176 smithi176.front.sepia.ceph.com smithi176 2024-01-08T07:02:30.116 DEBUG:teuthology.orchestra.run.smithi176:> sudo hostname smithi176 2024-01-08T07:02:30.236 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i -e 's/smithi176.front.sepia.ceph.com/smithi176/g' /etc/hosts 2024-01-08T07:02:30.266 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i -e 's/172.21.15.176/172.21.15.176/g' /etc/hosts 2024-01-08T07:02:30.334 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i -e 's/smithi176.front.sepia.ceph.com/smithi176/g' /etc/hostname 2024-01-08T07:02:30.403 DEBUG:teuthology.orchestra.run.smithi176:> sudo hostname smithi176 2024-01-08T07:02:30.471 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-01-08T07:02:30.528 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="Red Hat Enterprise Linux" 2024-01-08T07:02:30.528 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="8.6 (Ootpa)" 2024-01-08T07:02:30.528 INFO:teuthology.orchestra.run.smithi176.stdout:ID="rhel" 2024-01-08T07:02:30.528 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="fedora" 2024-01-08T07:02:30.528 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="8.6" 2024-01-08T07:02:30.528 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el8" 2024-01-08T07:02:30.529 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-08T07:02:30.529 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2024-01-08T07:02:30.529 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-08T07:02:30.529 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://www.redhat.com/" 2024-01-08T07:02:30.529 INFO:teuthology.orchestra.run.smithi176.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-08T07:02:30.529 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-08T07:02:30.529 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-08T07:02:30.529 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-08T07:02:30.529 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-08T07:02:30.529 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-08T07:02:30.529 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-08T07:02:30.529 INFO:teuthology.provision.fog.smithi176:Deploy complete! 2024-01-08T07:02:30.530 DEBUG:teuthology.parallel:result is None 2024-01-08T07:02:30.686 INFO:teuthology.lock.ops:Checking smithi026.front.sepia.ceph.com 2024-01-08T07:02:30.686 INFO:teuthology.lock.ops:New key found. Updating... 2024-01-08T07:02:30.715 INFO:teuthology.lock.ops:Checking smithi176.front.sepia.ceph.com 2024-01-08T07:02:30.716 INFO:teuthology.lock.ops:New key found. Updating... 2024-01-08T07:02:30.743 INFO:teuthology.lock.ops:Updating [smithi026.front.sepia.ceph.com]: set os type and version on server 2024-01-08T07:02:30.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-01-08T07:02:30.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:02:30.813 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-01-08T07:02:30.886 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-01-08T07:02:30.886 DEBUG:teuthology.orchestra.run.smithi026:> uname -m 2024-01-08T07:02:30.947 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2024-01-08T07:02:30.947 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2024-01-08T07:02:31.008 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="Red Hat Enterprise Linux" 2024-01-08T07:02:31.008 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="8.6 (Ootpa)" 2024-01-08T07:02:31.008 INFO:teuthology.orchestra.run.smithi026.stdout:ID="rhel" 2024-01-08T07:02:31.008 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE="fedora" 2024-01-08T07:02:31.008 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="8.6" 2024-01-08T07:02:31.008 INFO:teuthology.orchestra.run.smithi026.stdout:PLATFORM_ID="platform:el8" 2024-01-08T07:02:31.008 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-08T07:02:31.008 INFO:teuthology.orchestra.run.smithi026.stdout:ANSI_COLOR="0;31" 2024-01-08T07:02:31.008 INFO:teuthology.orchestra.run.smithi026.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-08T07:02:31.008 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://www.redhat.com/" 2024-01-08T07:02:31.009 INFO:teuthology.orchestra.run.smithi026.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-08T07:02:31.009 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-08T07:02:31.009 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-01-08T07:02:31.009 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-08T07:02:31.009 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-08T07:02:31.009 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-08T07:02:31.009 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-08T07:02:31.009 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2024-01-08T07:02:31.032 INFO:teuthology.lock.ops:Updating [smithi176.front.sepia.ceph.com]: set os type and version on server 2024-01-08T07:02:31.032 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-01-08T07:02:31.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-08T07:02:31.099 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-01-08T07:02:31.169 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-01-08T07:02:31.170 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2024-01-08T07:02:31.226 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2024-01-08T07:02:31.227 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-01-08T07:02:31.283 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="Red Hat Enterprise Linux" 2024-01-08T07:02:31.283 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="8.6 (Ootpa)" 2024-01-08T07:02:31.283 INFO:teuthology.orchestra.run.smithi176.stdout:ID="rhel" 2024-01-08T07:02:31.283 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="fedora" 2024-01-08T07:02:31.283 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="8.6" 2024-01-08T07:02:31.283 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el8" 2024-01-08T07:02:31.283 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-01-08T07:02:31.283 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2024-01-08T07:02:31.283 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-01-08T07:02:31.284 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://www.redhat.com/" 2024-01-08T07:02:31.284 INFO:teuthology.orchestra.run.smithi176.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-01-08T07:02:31.284 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-08T07:02:31.284 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-01-08T07:02:31.284 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-08T07:02:31.284 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-01-08T07:02:31.284 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-01-08T07:02:31.284 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-01-08T07:02:31.285 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-01-08T07:02:31.307 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-08T07:02:31.409 INFO:teuthology.dispatcher.supervisor:Running job 7508896 2024-01-08T07:02:31.410 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_01dd171a561ca33631bf0a38b105879779f72410/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2024-01-08_06:50:55-fs-wip-vshankar-testing-20240103.072409-testing-default-smithi/7508896 --name vshankar-2024-01-08_06:50:55-fs-wip-vshankar-testing-20240103.072409-testing-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} -- /home/teuthworker/archive/vshankar-2024-01-08_06:50:55-fs-wip-vshankar-testing-20240103.072409-testing-default-smithi/7508896/orig.config.yaml 2024-01-08T07:02:31.413 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2024-01-08_06:50:55-fs-wip-vshankar-testing-20240103.072409-testing-default-smithi/7508896 2024-01-08T07:02:31.413 INFO:teuthology.dispatcher.supervisor:Job PID: 3661087 2024-01-08T07:02:31.414 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2024-01-08T07:04:31.513 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-08T07:06:31.696 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-08T07:08:31.864 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-08T07:10:32.036 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-08T07:12:32.208 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-08T07:14:32.400 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-08T07:16:32.593 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-08T07:16:32.675 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1 2024-01-08T07:16:32.822 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2024-01-08T07:16:32.823 INFO:teuthology.nuke:Checking targets against current locks 2024-01-08T07:16:32.873 DEBUG:teuthology.nuke:shortname: smithi026 2024-01-08T07:16:32.873 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-08T07:16:32.896 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-08_06:50:55-fs-wip-vshankar-testing-20240103.072409-testing-default-smithi/7508896', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-08 06:53:51.632256', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA+vILhG6oDEOL54b3lmTb6jxvkJKIqJo8aM0M0l5yyvnkrF/Muy53hypEMsGQIopXCTl2xBXtKSdpVe1EoGtbY='} 2024-01-08T07:16:32.907 DEBUG:teuthology.nuke:shortname: smithi176 2024-01-08T07:16:32.908 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-08T07:16:32.935 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-01-08_06:50:55-fs-wip-vshankar-testing-20240103.072409-testing-default-smithi/7508896', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-08 06:53:51.633671', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD5c0S6YHp5a8P5zPBvJXLlkqSDyZZ1XJmgaS1P/tVlly4xNbbNR+k55ttl4P4rxme308/W4tKLt1LFBXIGBEI4='} 2024-01-08T07:16:32.981 INFO:teuthology.orchestra.console.smithi026:Power off 2024-01-08T07:16:32.981 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-08T07:16:33.068 INFO:teuthology.orchestra.console.smithi176:Power off 2024-01-08T07:16:33.068 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-08T07:16:33.087 DEBUG:teuthology.orchestra.console.smithi026:power off output: Chassis Power Control: Down/Off 2024-01-08T07:16:33.087 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-08T07:16:33.116 DEBUG:teuthology.orchestra.console.smithi176:power off output: Chassis Power Control: Down/Off 2024-01-08T07:16:33.117 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-08T07:16:33.132 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off 2024-01-08T07:16:33.161 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2024-01-08T07:16:33.233 INFO:teuthology.orchestra.console.smithi026:Power off completed 2024-01-08T07:16:33.262 INFO:teuthology.orchestra.console.smithi176:Power off completed 2024-01-08T07:16:33.394 INFO:teuthology.lock.ops:unlocked: smithi026.front.sepia.ceph.com 2024-01-08T07:16:33.414 INFO:teuthology.lock.ops:unlocked: smithi176.front.sepia.ceph.com