2024-03-03T09:02:37.116 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-03-03T09:02:37.117 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T09:02:37.287 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f241647cc10>, signals=[15]) 2024-03-03T09:02:37.287 INFO:teuthology.lock.ops:Start node 'smithi110.front.sepia.ceph.com' reimaging 2024-03-03T09:02:37.287 INFO:teuthology.lock.ops:Updating [smithi110.front.sepia.ceph.com]: reset os type and version on server 2024-03-03T09:02:37.287 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2024-03-03T09:02:37.305 INFO:teuthology.lock.ops:Start node 'smithi133.front.sepia.ceph.com' reimaging 2024-03-03T09:02:37.306 INFO:teuthology.lock.ops:Updating [smithi133.front.sepia.ceph.com]: reset os type and version on server 2024-03-03T09:02:37.306 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server 2024-03-03T09:02:37.330 INFO:teuthology.lock.ops:Start node 'smithi138.front.sepia.ceph.com' reimaging 2024-03-03T09:02:37.330 INFO:teuthology.lock.ops:Updating [smithi138.front.sepia.ceph.com]: reset os type and version on server 2024-03-03T09:02:37.330 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2024-03-03T09:02:37.419 DEBUG:teuthology.provision.fog.smithi110:Requesting image smithi_rhel_8.6 (ID 5) 2024-03-03T09:02:37.436 DEBUG:teuthology.provision.fog.smithi133:Requesting image smithi_rhel_8.6 (ID 5) 2024-03-03T09:02:37.481 INFO:teuthology.provision.fog.smithi110:Scheduling deploy of rhel 8.6 2024-03-03T09:02:37.485 INFO:teuthology.provision.fog.smithi133:Scheduling deploy of rhel 8.6 2024-03-03T09:02:37.486 DEBUG:teuthology.provision.fog.smithi138:Requesting image smithi_rhel_8.6 (ID 5) 2024-03-03T09:02:37.541 INFO:teuthology.provision.fog.smithi138:Scheduling deploy of rhel 8.6 2024-03-03T09:02:37.818 INFO:teuthology.orchestra.console.smithi110:Power off 2024-03-03T09:02:37.819 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-03T09:02:37.834 INFO:teuthology.orchestra.console.smithi138:Power off 2024-03-03T09:02:37.834 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-03T09:02:37.845 INFO:teuthology.orchestra.console.smithi133:Power off 2024-03-03T09:02:37.845 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-03T09:02:37.853 DEBUG:teuthology.orchestra.console.smithi110:power off output: Chassis Power Control: Down/Off 2024-03-03T09:02:37.854 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T09:02:37.862 DEBUG:teuthology.orchestra.console.smithi138:power off output: Chassis Power Control: Down/Off 2024-03-03T09:02:37.862 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T09:02:37.870 DEBUG:teuthology.orchestra.console.smithi133:power off output: Chassis Power Control: Down/Off 2024-03-03T09:02:37.870 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T09:02:37.877 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is off 2024-03-03T09:02:37.885 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is off 2024-03-03T09:02:37.890 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is off 2024-03-03T09:02:37.978 INFO:teuthology.orchestra.console.smithi110:Power off completed 2024-03-03T09:02:37.986 INFO:teuthology.orchestra.console.smithi138:Power off completed 2024-03-03T09:02:37.991 INFO:teuthology.orchestra.console.smithi133:Power off completed 2024-03-03T09:02:38.080 INFO:teuthology.orchestra.console.smithi110:Power on 2024-03-03T09:02:38.080 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-03T09:02:38.087 INFO:teuthology.orchestra.console.smithi138:Power on 2024-03-03T09:02:38.088 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-03T09:02:38.094 INFO:teuthology.orchestra.console.smithi133:Power on 2024-03-03T09:02:38.094 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-03T09:02:38.104 DEBUG:teuthology.orchestra.console.smithi110:power on output: Chassis Power Control: Up/On 2024-03-03T09:02:38.105 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T09:02:38.113 DEBUG:teuthology.orchestra.console.smithi133:power on output: Chassis Power Control: Up/On 2024-03-03T09:02:38.114 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T09:02:38.125 DEBUG:teuthology.orchestra.console.smithi138:power on output: Chassis Power Control: Up/On 2024-03-03T09:02:38.125 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T09:02:38.132 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is off 2024-03-03T09:02:38.134 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is off 2024-03-03T09:02:38.147 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is off 2024-03-03T09:02:42.137 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T09:02:42.145 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T09:02:42.151 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T09:02:42.248 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2024-03-03T09:02:42.253 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2024-03-03T09:02:42.258 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is off 2024-03-03T09:02:42.349 INFO:teuthology.orchestra.console.smithi110:Power on completed 2024-03-03T09:02:42.353 INFO:teuthology.orchestra.console.smithi133:Power on completed 2024-03-03T09:02:42.450 INFO:teuthology.provision.fog.smithi110:Waiting for deploy to finish 2024-03-03T09:02:42.454 INFO:teuthology.provision.fog.smithi133:Waiting for deploy to finish 2024-03-03T09:02:46.260 DEBUG:teuthology.orchestra.console.smithi138:pexpect command: ipmitool -H smithi138.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T09:02:46.483 DEBUG:teuthology.orchestra.console.smithi138:check power output: Chassis Power is on 2024-03-03T09:02:46.584 INFO:teuthology.orchestra.console.smithi138:Power on completed 2024-03-03T09:02:46.684 INFO:teuthology.provision.fog.smithi138:Waiting for deploy to finish 2024-03-03T09:05:58.923 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:06:01.980 WARNING:teuthology.provision.fog.smithi110:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-03-03T09:06:07.985 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:06:11.036 WARNING:teuthology.provision.fog.smithi110:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-03-03T09:06:13.981 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:06:17.040 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:06:17.052 WARNING:teuthology.provision.fog.smithi133:[Errno None] Unable to connect to port 22 on 172.21.15.133 2024-03-03T09:06:18.078 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:06:20.097 WARNING:teuthology.provision.fog.smithi110:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-03-03T09:06:21.152 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138 2024-03-03T09:06:23.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:06:26.100 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:06:26.108 WARNING:teuthology.provision.fog.smithi133:[Errno None] Unable to connect to port 22 on 172.21.15.133 2024-03-03T09:06:27.155 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:06:29.180 WARNING:teuthology.provision.fog.smithi110:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-03-03T09:06:30.236 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138 2024-03-03T09:06:32.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:06:35.164 WARNING:teuthology.provision.fog.smithi133:[Errno None] Unable to connect to port 22 on 172.21.15.133 2024-03-03T09:06:35.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:06:36.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:06:38.239 WARNING:teuthology.provision.fog.smithi110:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-03-03T09:06:39.296 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138 2024-03-03T09:06:41.167 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:06:44.224 WARNING:teuthology.provision.fog.smithi133:[Errno None] Unable to connect to port 22 on 172.21.15.133 2024-03-03T09:06:44.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:06:45.297 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:06:47.292 WARNING:teuthology.provision.fog.smithi110:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-03-03T09:06:48.352 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138 2024-03-03T09:06:50.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:06:53.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:06:53.308 WARNING:teuthology.provision.fog.smithi133:[Errno None] Unable to connect to port 22 on 172.21.15.133 2024-03-03T09:06:54.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:06:56.348 WARNING:teuthology.provision.fog.smithi110:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-03-03T09:06:57.436 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138 2024-03-03T09:06:59.311 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:07:02.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:07:02.364 WARNING:teuthology.provision.fog.smithi133:[Errno None] Unable to connect to port 22 on 172.21.15.133 2024-03-03T09:07:03.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:07:06.492 WARNING:teuthology.provision.fog.smithi138:[Errno None] Unable to connect to port 22 on 172.21.15.138 2024-03-03T09:07:08.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:07:12.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:07:12.572 WARNING:teuthology.provision.fog.smithi110:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-03-03T09:07:18.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:07:21.632 WARNING:teuthology.provision.fog.smithi110:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-03-03T09:07:26.908 WARNING:teuthology.provision.fog.smithi133:[Errno None] Unable to connect to port 22 on 172.21.15.133 2024-03-03T09:07:27.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:07:30.716 WARNING:teuthology.provision.fog.smithi110:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-03-03T09:07:32.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:07:35.968 WARNING:teuthology.provision.fog.smithi133:[Errno None] Unable to connect to port 22 on 172.21.15.133 2024-03-03T09:07:36.718 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:07:37.140 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-03T09:07:37.140 DEBUG:teuthology.orchestra.run.smithi110:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-03T09:07:41.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:07:45.020 WARNING:teuthology.provision.fog.smithi133:[Errno None] Unable to connect to port 22 on 172.21.15.133 2024-03-03T09:07:51.021 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:07:52.118 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-03T09:07:52.119 DEBUG:teuthology.orchestra.run.smithi133:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-03T09:07:57.880 INFO:teuthology.provision.fog.smithi110:Node is ready 2024-03-03T09:07:57.881 DEBUG:teuthology.orchestra.run.smithi110:> hostname 2024-03-03T09:07:57.904 INFO:teuthology.orchestra.run.smithi110.stdout:smithi045 2024-03-03T09:07:57.904 DEBUG:teuthology.orchestra.run.smithi110:> grep smithi045 /etc/hosts 2024-03-03T09:07:57.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-03T09:07:57.963 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i -e 's/smithi045/smithi110/g' /etc/hostname 2024-03-03T09:07:58.096 DEBUG:teuthology.orchestra.run.smithi110:> sudo hostname smithi110 2024-03-03T09:07:58.124 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2024-03-03T09:07:58.180 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="Red Hat Enterprise Linux" 2024-03-03T09:07:58.180 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="8.6 (Ootpa)" 2024-03-03T09:07:58.180 INFO:teuthology.orchestra.run.smithi110.stdout:ID="rhel" 2024-03-03T09:07:58.181 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="fedora" 2024-03-03T09:07:58.181 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="8.6" 2024-03-03T09:07:58.181 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el8" 2024-03-03T09:07:58.181 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-03T09:07:58.181 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31" 2024-03-03T09:07:58.181 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-03T09:07:58.181 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://www.redhat.com/" 2024-03-03T09:07:58.181 INFO:teuthology.orchestra.run.smithi110.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-03T09:07:58.181 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-03T09:07:58.181 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-03T09:07:58.181 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-03T09:07:58.181 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-03T09:07:58.181 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-03T09:07:58.182 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-03T09:07:58.182 INFO:teuthology.provision.fog.smithi110:Deploy complete! 2024-03-03T09:07:58.183 DEBUG:teuthology.parallel:result is None 2024-03-03T09:08:07.881 INFO:teuthology.provision.fog.smithi133:Node is ready 2024-03-03T09:08:07.881 DEBUG:teuthology.orchestra.run.smithi133:> hostname 2024-03-03T09:08:07.944 INFO:teuthology.orchestra.run.smithi133.stdout:smithi133.front.sepia.ceph.com 2024-03-03T09:08:07.944 DEBUG:teuthology.orchestra.run.smithi133:> grep smithi133.front.sepia.ceph.com /etc/hosts 2024-03-03T09:08:08.003 INFO:teuthology.orchestra.run.smithi133.stdout:172.21.15.133 smithi133.front.sepia.ceph.com smithi133 2024-03-03T09:08:08.004 DEBUG:teuthology.orchestra.run.smithi133:> sudo hostname smithi133 2024-03-03T09:08:08.131 DEBUG:teuthology.orchestra.run.smithi133:> sudo sed -i -e 's/smithi133.front.sepia.ceph.com/smithi133/g' /etc/hosts 2024-03-03T09:08:08.165 DEBUG:teuthology.orchestra.run.smithi133:> sudo sed -i -e 's/172.21.15.133/172.21.15.133/g' /etc/hosts 2024-03-03T09:08:08.235 DEBUG:teuthology.orchestra.run.smithi133:> sudo sed -i -e 's/smithi133.front.sepia.ceph.com/smithi133/g' /etc/hostname 2024-03-03T09:08:08.303 DEBUG:teuthology.orchestra.run.smithi133:> sudo hostname smithi133 2024-03-03T09:08:08.372 DEBUG:teuthology.orchestra.run.smithi133:> cat /etc/os-release 2024-03-03T09:08:08.430 INFO:teuthology.orchestra.run.smithi133.stdout:NAME="Red Hat Enterprise Linux" 2024-03-03T09:08:08.430 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION="8.6 (Ootpa)" 2024-03-03T09:08:08.430 INFO:teuthology.orchestra.run.smithi133.stdout:ID="rhel" 2024-03-03T09:08:08.430 INFO:teuthology.orchestra.run.smithi133.stdout:ID_LIKE="fedora" 2024-03-03T09:08:08.430 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_ID="8.6" 2024-03-03T09:08:08.430 INFO:teuthology.orchestra.run.smithi133.stdout:PLATFORM_ID="platform:el8" 2024-03-03T09:08:08.430 INFO:teuthology.orchestra.run.smithi133.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-03T09:08:08.431 INFO:teuthology.orchestra.run.smithi133.stdout:ANSI_COLOR="0;31" 2024-03-03T09:08:08.431 INFO:teuthology.orchestra.run.smithi133.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-03T09:08:08.431 INFO:teuthology.orchestra.run.smithi133.stdout:HOME_URL="https://www.redhat.com/" 2024-03-03T09:08:08.431 INFO:teuthology.orchestra.run.smithi133.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-03T09:08:08.431 INFO:teuthology.orchestra.run.smithi133.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-03T09:08:08.431 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-03T09:08:08.431 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-03T09:08:08.431 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-03T09:08:08.431 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-03T09:08:08.431 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-03T09:08:08.431 INFO:teuthology.provision.fog.smithi133:Deploy complete! 2024-03-03T09:08:08.432 DEBUG:teuthology.parallel:result is None 2024-03-03T09:08:12.499 WARNING:teuthology.provision.fog.smithi138:timed out 2024-03-03T09:08:18.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:08:18.736 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-03T09:08:18.737 DEBUG:teuthology.orchestra.run.smithi138:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-03T09:08:19.001 INFO:teuthology.provision.fog.smithi138:Node is ready 2024-03-03T09:08:19.001 DEBUG:teuthology.orchestra.run.smithi138:> hostname 2024-03-03T09:08:19.019 INFO:teuthology.orchestra.run.smithi138.stdout:smithi138 2024-03-03T09:08:19.020 DEBUG:teuthology.orchestra.run.smithi138:> grep smithi138 /etc/hosts 2024-03-03T09:08:19.078 INFO:teuthology.orchestra.run.smithi138.stdout:172.21.15.138 smithi138.front.sepia.ceph.com smithi138 2024-03-03T09:08:19.078 DEBUG:teuthology.orchestra.run.smithi138:> sudo hostname smithi138 2024-03-03T09:08:19.185 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i -e 's/smithi138/smithi138/g' /etc/hosts 2024-03-03T09:08:19.213 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i -e 's/172.21.15.138/172.21.15.138/g' /etc/hosts 2024-03-03T09:08:19.281 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i -e 's/smithi138/smithi138/g' /etc/hostname 2024-03-03T09:08:19.348 DEBUG:teuthology.orchestra.run.smithi138:> sudo hostname smithi138 2024-03-03T09:08:19.414 DEBUG:teuthology.orchestra.run.smithi138:> cat /etc/os-release 2024-03-03T09:08:19.471 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="Red Hat Enterprise Linux" 2024-03-03T09:08:19.472 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="8.6 (Ootpa)" 2024-03-03T09:08:19.472 INFO:teuthology.orchestra.run.smithi138.stdout:ID="rhel" 2024-03-03T09:08:19.472 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE="fedora" 2024-03-03T09:08:19.472 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="8.6" 2024-03-03T09:08:19.472 INFO:teuthology.orchestra.run.smithi138.stdout:PLATFORM_ID="platform:el8" 2024-03-03T09:08:19.472 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-03T09:08:19.472 INFO:teuthology.orchestra.run.smithi138.stdout:ANSI_COLOR="0;31" 2024-03-03T09:08:19.472 INFO:teuthology.orchestra.run.smithi138.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-03T09:08:19.472 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://www.redhat.com/" 2024-03-03T09:08:19.472 INFO:teuthology.orchestra.run.smithi138.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-03T09:08:19.472 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-03T09:08:19.473 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-03T09:08:19.473 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-03T09:08:19.473 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-03T09:08:19.473 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-03T09:08:19.473 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-03T09:08:19.473 INFO:teuthology.provision.fog.smithi138:Deploy complete! 2024-03-03T09:08:19.474 DEBUG:teuthology.parallel:result is None 2024-03-03T09:08:19.664 INFO:teuthology.lock.ops:Checking smithi110.front.sepia.ceph.com 2024-03-03T09:08:19.665 INFO:teuthology.lock.ops:New key found. Updating... 2024-03-03T09:08:19.689 INFO:teuthology.lock.ops:Checking smithi133.front.sepia.ceph.com 2024-03-03T09:08:19.689 INFO:teuthology.lock.ops:New key found. Updating... 2024-03-03T09:08:19.705 INFO:teuthology.lock.ops:Checking smithi138.front.sepia.ceph.com 2024-03-03T09:08:19.705 INFO:teuthology.lock.ops:New key found. Updating... 2024-03-03T09:08:19.723 INFO:teuthology.lock.ops:Updating [smithi110.front.sepia.ceph.com]: set os type and version on server 2024-03-03T09:08:19.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-03T09:08:19.724 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:08:19.803 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-03-03T09:08:19.945 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-03-03T09:08:19.946 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2024-03-03T09:08:20.005 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2024-03-03T09:08:20.006 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2024-03-03T09:08:20.063 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="Red Hat Enterprise Linux" 2024-03-03T09:08:20.063 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="8.6 (Ootpa)" 2024-03-03T09:08:20.063 INFO:teuthology.orchestra.run.smithi110.stdout:ID="rhel" 2024-03-03T09:08:20.063 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="fedora" 2024-03-03T09:08:20.063 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="8.6" 2024-03-03T09:08:20.063 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el8" 2024-03-03T09:08:20.063 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-03T09:08:20.063 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31" 2024-03-03T09:08:20.063 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-03T09:08:20.063 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://www.redhat.com/" 2024-03-03T09:08:20.063 INFO:teuthology.orchestra.run.smithi110.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-03T09:08:20.064 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-03T09:08:20.064 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-03-03T09:08:20.064 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-03T09:08:20.064 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-03T09:08:20.064 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-03T09:08:20.064 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-03T09:08:20.064 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2024-03-03T09:08:20.089 INFO:teuthology.lock.ops:Updating [smithi133.front.sepia.ceph.com]: set os type and version on server 2024-03-03T09:08:20.090 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-03-03T09:08:20.090 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:08:20.162 DEBUG:teuthology.orchestra.run.smithi133:> true 2024-03-03T09:08:20.237 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-03-03T09:08:20.238 DEBUG:teuthology.orchestra.run.smithi133:> uname -m 2024-03-03T09:08:20.298 INFO:teuthology.orchestra.run.smithi133.stdout:x86_64 2024-03-03T09:08:20.298 DEBUG:teuthology.orchestra.run.smithi133:> cat /etc/os-release 2024-03-03T09:08:20.358 INFO:teuthology.orchestra.run.smithi133.stdout:NAME="Red Hat Enterprise Linux" 2024-03-03T09:08:20.358 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION="8.6 (Ootpa)" 2024-03-03T09:08:20.358 INFO:teuthology.orchestra.run.smithi133.stdout:ID="rhel" 2024-03-03T09:08:20.358 INFO:teuthology.orchestra.run.smithi133.stdout:ID_LIKE="fedora" 2024-03-03T09:08:20.358 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_ID="8.6" 2024-03-03T09:08:20.358 INFO:teuthology.orchestra.run.smithi133.stdout:PLATFORM_ID="platform:el8" 2024-03-03T09:08:20.358 INFO:teuthology.orchestra.run.smithi133.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-03T09:08:20.358 INFO:teuthology.orchestra.run.smithi133.stdout:ANSI_COLOR="0;31" 2024-03-03T09:08:20.358 INFO:teuthology.orchestra.run.smithi133.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-03T09:08:20.358 INFO:teuthology.orchestra.run.smithi133.stdout:HOME_URL="https://www.redhat.com/" 2024-03-03T09:08:20.358 INFO:teuthology.orchestra.run.smithi133.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-03T09:08:20.358 INFO:teuthology.orchestra.run.smithi133.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-03T09:08:20.358 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-03-03T09:08:20.359 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-03T09:08:20.359 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-03T09:08:20.359 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-03T09:08:20.359 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-03T09:08:20.359 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server 2024-03-03T09:08:20.381 INFO:teuthology.lock.ops:Updating [smithi138.front.sepia.ceph.com]: set os type and version on server 2024-03-03T09:08:20.381 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-03T09:08:20.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T09:08:20.475 DEBUG:teuthology.orchestra.run.smithi138:> true 2024-03-03T09:08:20.542 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-03-03T09:08:20.542 DEBUG:teuthology.orchestra.run.smithi138:> uname -m 2024-03-03T09:08:20.601 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64 2024-03-03T09:08:20.601 DEBUG:teuthology.orchestra.run.smithi138:> cat /etc/os-release 2024-03-03T09:08:20.659 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="Red Hat Enterprise Linux" 2024-03-03T09:08:20.659 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="8.6 (Ootpa)" 2024-03-03T09:08:20.659 INFO:teuthology.orchestra.run.smithi138.stdout:ID="rhel" 2024-03-03T09:08:20.659 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE="fedora" 2024-03-03T09:08:20.659 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="8.6" 2024-03-03T09:08:20.660 INFO:teuthology.orchestra.run.smithi138.stdout:PLATFORM_ID="platform:el8" 2024-03-03T09:08:20.660 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-03T09:08:20.660 INFO:teuthology.orchestra.run.smithi138.stdout:ANSI_COLOR="0;31" 2024-03-03T09:08:20.660 INFO:teuthology.orchestra.run.smithi138.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-03T09:08:20.660 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://www.redhat.com/" 2024-03-03T09:08:20.660 INFO:teuthology.orchestra.run.smithi138.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-03T09:08:20.660 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-03T09:08:20.660 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-03-03T09:08:20.661 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-03T09:08:20.661 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-03T09:08:20.661 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-03T09:08:20.661 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-03T09:08:20.661 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2024-03-03T09:08:20.689 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T09:08:20.782 INFO:teuthology.dispatcher.supervisor:Running job 7580161 2024-03-03T09:08:20.783 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/bin/teuthology -v --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2024-03-02_17:47:35-fs-reef-distro-default-smithi/7580161 --name pdonnell-2024-03-02_17:47:35-fs-reef-distro-default-smithi --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} -- /home/teuthworker/archive/pdonnell-2024-03-02_17:47:35-fs-reef-distro-default-smithi/7580161/orig.config.yaml 2024-03-03T09:08:20.786 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/pdonnell-2024-03-02_17:47:35-fs-reef-distro-default-smithi/7580161 2024-03-03T09:08:20.786 INFO:teuthology.dispatcher.supervisor:Job PID: 222424 2024-03-03T09:08:20.786 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2024-03-03T09:10:20.884 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T09:12:21.060 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T09:14:21.232 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T09:16:21.400 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T09:18:21.576 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T09:20:21.748 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T09:22:21.916 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T09:24:22.099 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T09:26:22.272 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T09:28:22.446 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T09:30:22.628 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T09:32:22.808 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T09:32:22.876 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1