2024-03-03T07:48:21.305 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-03-03T07:48:21.306 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T07:48:21.474 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f091cfc4c10>, signals=[15]) 2024-03-03T07:48:21.475 INFO:teuthology.lock.ops:Start node 'smithi029.front.sepia.ceph.com' reimaging 2024-03-03T07:48:21.475 INFO:teuthology.lock.ops:Updating [smithi029.front.sepia.ceph.com]: reset os type and version on server 2024-03-03T07:48:21.475 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-03-03T07:48:21.497 INFO:teuthology.lock.ops:Start node 'smithi076.front.sepia.ceph.com' reimaging 2024-03-03T07:48:21.498 INFO:teuthology.lock.ops:Updating [smithi076.front.sepia.ceph.com]: reset os type and version on server 2024-03-03T07:48:21.498 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2024-03-03T07:48:21.522 INFO:teuthology.lock.ops:Start node 'smithi188.front.sepia.ceph.com' reimaging 2024-03-03T07:48:21.523 INFO:teuthology.lock.ops:Updating [smithi188.front.sepia.ceph.com]: reset os type and version on server 2024-03-03T07:48:21.523 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2024-03-03T07:48:21.621 DEBUG:teuthology.provision.fog.smithi029:Requesting image smithi_rhel_8.6 (ID 5) 2024-03-03T07:48:21.648 DEBUG:teuthology.provision.fog.smithi076:Requesting image smithi_rhel_8.6 (ID 5) 2024-03-03T07:48:21.658 DEBUG:teuthology.provision.fog.smithi188:Requesting image smithi_rhel_8.6 (ID 5) 2024-03-03T07:48:21.663 INFO:teuthology.provision.fog.smithi029:Scheduling deploy of rhel 8.6 2024-03-03T07:48:21.688 INFO:teuthology.provision.fog.smithi076:Scheduling deploy of rhel 8.6 2024-03-03T07:48:21.697 INFO:teuthology.provision.fog.smithi188:Scheduling deploy of rhel 8.6 2024-03-03T07:48:22.122 INFO:teuthology.orchestra.console.smithi188:Power off 2024-03-03T07:48:22.123 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-03T07:48:22.140 INFO:teuthology.orchestra.console.smithi029:Power off 2024-03-03T07:48:22.140 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-03T07:48:22.155 INFO:teuthology.orchestra.console.smithi076:Power off 2024-03-03T07:48:22.156 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-03T07:48:22.237 DEBUG:teuthology.orchestra.console.smithi188:power off output: Chassis Power Control: Down/Off 2024-03-03T07:48:22.238 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T07:48:22.246 DEBUG:teuthology.orchestra.console.smithi076:power off output: Chassis Power Control: Down/Off 2024-03-03T07:48:22.246 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T07:48:22.255 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2024-03-03T07:48:22.255 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T07:48:22.276 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2024-03-03T07:48:22.291 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-03-03T07:48:22.332 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-03-03T07:48:26.280 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T07:48:26.291 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T07:48:26.333 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T07:48:26.390 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2024-03-03T07:48:26.401 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-03-03T07:48:26.444 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-03-03T07:48:30.392 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T07:48:30.402 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T07:48:30.445 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T07:48:30.502 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off 2024-03-03T07:48:30.511 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is off 2024-03-03T07:48:30.553 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2024-03-03T07:48:30.603 INFO:teuthology.orchestra.console.smithi076:Power off completed 2024-03-03T07:48:30.611 INFO:teuthology.orchestra.console.smithi188:Power off completed 2024-03-03T07:48:30.655 INFO:teuthology.orchestra.console.smithi029:Power off completed 2024-03-03T07:48:30.704 INFO:teuthology.orchestra.console.smithi076:Power on 2024-03-03T07:48:30.705 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-03T07:48:30.713 INFO:teuthology.orchestra.console.smithi188:Power on 2024-03-03T07:48:30.713 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-03T07:48:30.727 DEBUG:teuthology.orchestra.console.smithi076:power on output: Chassis Power Control: Up/On 2024-03-03T07:48:30.727 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T07:48:30.735 DEBUG:teuthology.orchestra.console.smithi188:power on output: Chassis Power Control: Up/On 2024-03-03T07:48:30.735 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T07:48:30.750 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off 2024-03-03T07:48:30.756 INFO:teuthology.orchestra.console.smithi029:Power on 2024-03-03T07:48:30.756 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-03T07:48:30.763 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is off 2024-03-03T07:48:30.779 DEBUG:teuthology.orchestra.console.smithi029:power on output: Chassis Power Control: Up/On 2024-03-03T07:48:30.779 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T07:48:30.801 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2024-03-03T07:48:34.755 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T07:48:34.765 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T07:48:34.802 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T07:48:34.864 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off 2024-03-03T07:48:34.874 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is off 2024-03-03T07:48:34.911 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2024-03-03T07:48:38.868 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T07:48:38.877 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T07:48:38.912 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-03T07:48:38.985 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-03-03T07:48:39.021 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2024-03-03T07:48:39.024 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-03-03T07:48:39.087 INFO:teuthology.orchestra.console.smithi188:Power on completed 2024-03-03T07:48:39.121 INFO:teuthology.orchestra.console.smithi076:Power on completed 2024-03-03T07:48:39.124 INFO:teuthology.orchestra.console.smithi029:Power on completed 2024-03-03T07:48:39.188 INFO:teuthology.provision.fog.smithi188:Waiting for deploy to finish 2024-03-03T07:48:39.223 INFO:teuthology.provision.fog.smithi076:Waiting for deploy to finish 2024-03-03T07:48:39.226 INFO:teuthology.provision.fog.smithi029:Waiting for deploy to finish 2024-03-03T07:51:25.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:51:25.158 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:51:25.463 ERROR:teuthology.orchestra.connection:Error authenticating with smithi076.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-03-03T07:51:28.220 WARNING:teuthology.provision.fog.smithi188:[Errno None] Unable to connect to port 22 on 172.21.15.188 2024-03-03T07:51:34.222 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:51:37.276 WARNING:teuthology.provision.fog.smithi188:[Errno None] Unable to connect to port 22 on 172.21.15.188 2024-03-03T07:51:43.277 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:51:46.332 WARNING:teuthology.provision.fog.smithi188:[Errno None] Unable to connect to port 22 on 172.21.15.188 2024-03-03T07:51:52.334 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:51:55.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:51:55.388 WARNING:teuthology.provision.fog.smithi188:[Errno None] Unable to connect to port 22 on 172.21.15.188 2024-03-03T07:51:58.336 WARNING:teuthology.provision.fog.smithi029:[Errno None] Unable to connect to port 22 on 172.21.15.29 2024-03-03T07:52:01.389 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:52:04.337 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:52:04.444 WARNING:teuthology.provision.fog.smithi188:[Errno None] Unable to connect to port 22 on 172.21.15.188 2024-03-03T07:52:07.388 WARNING:teuthology.provision.fog.smithi029:[Errno None] Unable to connect to port 22 on 172.21.15.29 2024-03-03T07:52:10.446 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:52:13.389 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:52:13.504 WARNING:teuthology.provision.fog.smithi188:[Errno None] Unable to connect to port 22 on 172.21.15.188 2024-03-03T07:52:16.444 WARNING:teuthology.provision.fog.smithi029:[Errno None] Unable to connect to port 22 on 172.21.15.29 2024-03-03T07:52:19.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:52:22.445 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:52:25.500 WARNING:teuthology.provision.fog.smithi029:[Errno None] Unable to connect to port 22 on 172.21.15.29 2024-03-03T07:52:29.465 WARNING:teuthology.provision.fog.smithi076:timed out 2024-03-03T07:52:31.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:52:34.556 WARNING:teuthology.provision.fog.smithi029:[Errno None] Unable to connect to port 22 on 172.21.15.29 2024-03-03T07:52:35.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:52:40.557 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:53:19.507 WARNING:teuthology.provision.fog.smithi188:timed out 2024-03-03T07:53:25.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:53:25.756 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-03T07:53:25.757 DEBUG:teuthology.orchestra.run.smithi188:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-03T07:53:26.026 INFO:teuthology.provision.fog.smithi188:Node is ready 2024-03-03T07:53:26.026 DEBUG:teuthology.orchestra.run.smithi188:> hostname 2024-03-03T07:53:26.043 INFO:teuthology.orchestra.run.smithi188.stdout:smithi188 2024-03-03T07:53:26.043 DEBUG:teuthology.orchestra.run.smithi188:> grep smithi188 /etc/hosts 2024-03-03T07:53:26.101 INFO:teuthology.orchestra.run.smithi188.stdout:172.21.15.188 smithi188.front.sepia.ceph.com smithi188 2024-03-03T07:53:26.101 DEBUG:teuthology.orchestra.run.smithi188:> sudo hostname smithi188 2024-03-03T07:53:26.211 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i -e 's/smithi188/smithi188/g' /etc/hosts 2024-03-03T07:53:26.239 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i -e 's/172.21.15.188/172.21.15.188/g' /etc/hosts 2024-03-03T07:53:26.306 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i -e 's/smithi188/smithi188/g' /etc/hostname 2024-03-03T07:53:26.377 DEBUG:teuthology.orchestra.run.smithi188:> sudo hostname smithi188 2024-03-03T07:53:26.455 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2024-03-03T07:53:26.523 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="Red Hat Enterprise Linux" 2024-03-03T07:53:26.523 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="8.6 (Ootpa)" 2024-03-03T07:53:26.523 INFO:teuthology.orchestra.run.smithi188.stdout:ID="rhel" 2024-03-03T07:53:26.523 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="fedora" 2024-03-03T07:53:26.523 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="8.6" 2024-03-03T07:53:26.524 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el8" 2024-03-03T07:53:26.524 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-03T07:53:26.524 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2024-03-03T07:53:26.524 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-03T07:53:26.524 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://www.redhat.com/" 2024-03-03T07:53:26.524 INFO:teuthology.orchestra.run.smithi188.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-03T07:53:26.524 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-03T07:53:26.524 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-03T07:53:26.524 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-03T07:53:26.525 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-03T07:53:26.525 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-03T07:53:26.525 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-03T07:53:26.526 INFO:teuthology.provision.fog.smithi188:Deploy complete! 2024-03-03T07:53:26.528 DEBUG:teuthology.parallel:result is None 2024-03-03T07:53:35.468 WARNING:teuthology.provision.fog.smithi076:timed out 2024-03-03T07:53:40.559 WARNING:teuthology.provision.fog.smithi029:timed out 2024-03-03T07:53:41.470 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:53:41.752 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-03T07:53:41.753 DEBUG:teuthology.orchestra.run.smithi076:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-03T07:53:42.019 INFO:teuthology.provision.fog.smithi076:Node is ready 2024-03-03T07:53:42.020 DEBUG:teuthology.orchestra.run.smithi076:> hostname 2024-03-03T07:53:42.039 INFO:teuthology.orchestra.run.smithi076.stdout:smithi076.front.sepia.ceph.com 2024-03-03T07:53:42.039 DEBUG:teuthology.orchestra.run.smithi076:> grep smithi076.front.sepia.ceph.com /etc/hosts 2024-03-03T07:53:42.099 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76 smithi076.front.sepia.ceph.com smithi076 2024-03-03T07:53:42.100 DEBUG:teuthology.orchestra.run.smithi076:> sudo hostname smithi076 2024-03-03T07:53:42.216 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i -e 's/smithi076.front.sepia.ceph.com/smithi076/g' /etc/hosts 2024-03-03T07:53:42.246 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i -e 's/172.21.15.76/172.21.15.76/g' /etc/hosts 2024-03-03T07:53:42.315 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i -e 's/smithi076.front.sepia.ceph.com/smithi076/g' /etc/hostname 2024-03-03T07:53:42.383 DEBUG:teuthology.orchestra.run.smithi076:> sudo hostname smithi076 2024-03-03T07:53:42.450 DEBUG:teuthology.orchestra.run.smithi076:> cat /etc/os-release 2024-03-03T07:53:42.508 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="Red Hat Enterprise Linux" 2024-03-03T07:53:42.508 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="8.6 (Ootpa)" 2024-03-03T07:53:42.508 INFO:teuthology.orchestra.run.smithi076.stdout:ID="rhel" 2024-03-03T07:53:42.508 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="fedora" 2024-03-03T07:53:42.509 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="8.6" 2024-03-03T07:53:42.509 INFO:teuthology.orchestra.run.smithi076.stdout:PLATFORM_ID="platform:el8" 2024-03-03T07:53:42.509 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-03T07:53:42.509 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31" 2024-03-03T07:53:42.509 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-03T07:53:42.509 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://www.redhat.com/" 2024-03-03T07:53:42.509 INFO:teuthology.orchestra.run.smithi076.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-03T07:53:42.509 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-03T07:53:42.509 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-03T07:53:42.509 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-03T07:53:42.509 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-03T07:53:42.509 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-03T07:53:42.510 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-03T07:53:42.510 INFO:teuthology.provision.fog.smithi076:Deploy complete! 2024-03-03T07:53:42.511 DEBUG:teuthology.parallel:result is None 2024-03-03T07:53:46.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:53:46.809 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-03T07:53:46.810 DEBUG:teuthology.orchestra.run.smithi029:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-03T07:53:47.116 INFO:teuthology.provision.fog.smithi029:Node is ready 2024-03-03T07:53:47.117 DEBUG:teuthology.orchestra.run.smithi029:> hostname 2024-03-03T07:53:47.138 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029.front.sepia.ceph.com 2024-03-03T07:53:47.138 DEBUG:teuthology.orchestra.run.smithi029:> grep smithi029.front.sepia.ceph.com /etc/hosts 2024-03-03T07:53:47.200 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29 smithi029.front.sepia.ceph.com smithi029 2024-03-03T07:53:47.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo hostname smithi029 2024-03-03T07:53:47.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i -e 's/smithi029.front.sepia.ceph.com/smithi029/g' /etc/hosts 2024-03-03T07:53:47.357 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i -e 's/172.21.15.29/172.21.15.29/g' /etc/hosts 2024-03-03T07:53:47.432 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i -e 's/smithi029.front.sepia.ceph.com/smithi029/g' /etc/hostname 2024-03-03T07:53:47.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo hostname smithi029 2024-03-03T07:53:47.582 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-03-03T07:53:47.644 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="Red Hat Enterprise Linux" 2024-03-03T07:53:47.644 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="8.6 (Ootpa)" 2024-03-03T07:53:47.644 INFO:teuthology.orchestra.run.smithi029.stdout:ID="rhel" 2024-03-03T07:53:47.644 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="fedora" 2024-03-03T07:53:47.644 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="8.6" 2024-03-03T07:53:47.644 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el8" 2024-03-03T07:53:47.644 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-03T07:53:47.644 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-03-03T07:53:47.644 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-03T07:53:47.644 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://www.redhat.com/" 2024-03-03T07:53:47.645 INFO:teuthology.orchestra.run.smithi029.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-03T07:53:47.645 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-03T07:53:47.645 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-03T07:53:47.645 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-03T07:53:47.645 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-03T07:53:47.645 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-03T07:53:47.645 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-03T07:53:47.645 INFO:teuthology.provision.fog.smithi029:Deploy complete! 2024-03-03T07:53:47.646 DEBUG:teuthology.parallel:result is None 2024-03-03T07:53:47.871 INFO:teuthology.lock.ops:Checking smithi029.front.sepia.ceph.com 2024-03-03T07:53:47.871 INFO:teuthology.lock.ops:New key found. Updating... 2024-03-03T07:53:47.892 INFO:teuthology.lock.ops:Checking smithi076.front.sepia.ceph.com 2024-03-03T07:53:47.892 INFO:teuthology.lock.ops:New key found. Updating... 2024-03-03T07:53:47.908 INFO:teuthology.lock.ops:Checking smithi188.front.sepia.ceph.com 2024-03-03T07:53:47.908 INFO:teuthology.lock.ops:New key found. Updating... 2024-03-03T07:53:47.926 INFO:teuthology.lock.ops:Updating [smithi029.front.sepia.ceph.com]: set os type and version on server 2024-03-03T07:53:47.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-03T07:53:47.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:53:48.005 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-03-03T07:53:48.081 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-03T07:53:48.081 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-03-03T07:53:48.142 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-03-03T07:53:48.142 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-03-03T07:53:48.203 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="Red Hat Enterprise Linux" 2024-03-03T07:53:48.203 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="8.6 (Ootpa)" 2024-03-03T07:53:48.203 INFO:teuthology.orchestra.run.smithi029.stdout:ID="rhel" 2024-03-03T07:53:48.203 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="fedora" 2024-03-03T07:53:48.203 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="8.6" 2024-03-03T07:53:48.203 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el8" 2024-03-03T07:53:48.203 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-03T07:53:48.204 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-03-03T07:53:48.204 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-03T07:53:48.204 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://www.redhat.com/" 2024-03-03T07:53:48.204 INFO:teuthology.orchestra.run.smithi029.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-03T07:53:48.204 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-03T07:53:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-03T07:53:48.204 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-03T07:53:48.204 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-03T07:53:48.204 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-03T07:53:48.204 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-03T07:53:48.205 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-03-03T07:53:48.227 INFO:teuthology.lock.ops:Updating [smithi076.front.sepia.ceph.com]: set os type and version on server 2024-03-03T07:53:48.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-03T07:53:48.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:53:48.301 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-03-03T07:53:48.375 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-03T07:53:48.376 DEBUG:teuthology.orchestra.run.smithi076:> uname -m 2024-03-03T07:53:48.437 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64 2024-03-03T07:53:48.438 DEBUG:teuthology.orchestra.run.smithi076:> cat /etc/os-release 2024-03-03T07:53:48.498 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="Red Hat Enterprise Linux" 2024-03-03T07:53:48.498 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="8.6 (Ootpa)" 2024-03-03T07:53:48.498 INFO:teuthology.orchestra.run.smithi076.stdout:ID="rhel" 2024-03-03T07:53:48.498 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="fedora" 2024-03-03T07:53:48.499 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="8.6" 2024-03-03T07:53:48.499 INFO:teuthology.orchestra.run.smithi076.stdout:PLATFORM_ID="platform:el8" 2024-03-03T07:53:48.499 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-03T07:53:48.499 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31" 2024-03-03T07:53:48.499 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-03T07:53:48.499 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://www.redhat.com/" 2024-03-03T07:53:48.499 INFO:teuthology.orchestra.run.smithi076.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-03T07:53:48.499 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-03T07:53:48.499 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-03T07:53:48.499 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-03T07:53:48.499 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-03T07:53:48.499 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-03T07:53:48.499 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-03T07:53:48.500 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2024-03-03T07:53:48.525 INFO:teuthology.lock.ops:Updating [smithi188.front.sepia.ceph.com]: set os type and version on server 2024-03-03T07:53:48.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-03-03T07:53:48.526 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-03T07:53:48.597 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-03-03T07:53:48.693 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-03-03T07:53:48.693 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2024-03-03T07:53:48.751 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2024-03-03T07:53:48.752 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2024-03-03T07:53:48.813 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="Red Hat Enterprise Linux" 2024-03-03T07:53:48.813 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="8.6 (Ootpa)" 2024-03-03T07:53:48.813 INFO:teuthology.orchestra.run.smithi188.stdout:ID="rhel" 2024-03-03T07:53:48.813 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="fedora" 2024-03-03T07:53:48.813 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="8.6" 2024-03-03T07:53:48.813 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el8" 2024-03-03T07:53:48.813 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-03T07:53:48.813 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2024-03-03T07:53:48.813 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-03T07:53:48.813 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://www.redhat.com/" 2024-03-03T07:53:48.814 INFO:teuthology.orchestra.run.smithi188.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-03T07:53:48.814 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-03T07:53:48.814 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-03-03T07:53:48.814 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-03T07:53:48.814 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-03T07:53:48.814 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-03T07:53:48.814 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-03T07:53:48.815 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2024-03-03T07:53:48.834 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T07:53:48.938 INFO:teuthology.dispatcher.supervisor:Running job 7580075 2024-03-03T07:53:48.938 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/bin/teuthology -v --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2024-03-02_17:47:35-fs-reef-distro-default-smithi/7580075 --name pdonnell-2024-03-02_17:47:35-fs-reef-distro-default-smithi --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} -- /home/teuthworker/archive/pdonnell-2024-03-02_17:47:35-fs-reef-distro-default-smithi/7580075/orig.config.yaml 2024-03-03T07:53:48.941 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/pdonnell-2024-03-02_17:47:35-fs-reef-distro-default-smithi/7580075 2024-03-03T07:53:48.941 INFO:teuthology.dispatcher.supervisor:Job PID: 4039275 2024-03-03T07:53:48.941 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2024-03-03T07:55:48.943 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T07:57:49.024 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T07:59:49.090 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T08:01:49.163 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T08:03:49.241 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T08:05:49.314 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T08:07:49.392 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T08:09:49.465 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T08:11:49.539 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T08:13:49.604 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T08:15:49.677 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T08:17:49.753 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T08:19:49.850 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T08:21:49.919 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T08:23:49.999 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T08:25:50.071 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T08:27:50.142 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-03T08:27:50.214 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1