2024-04-18T03:06:03.308 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-18T03:06:03.309 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-18T03:06:03.492 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f90997a4c10>, signals=[15]) 2024-04-18T03:06:03.492 INFO:teuthology.lock.ops:Start node 'smithi084.front.sepia.ceph.com' reimaging 2024-04-18T03:06:03.493 INFO:teuthology.lock.ops:Updating [smithi084.front.sepia.ceph.com]: reset os type and version on server 2024-04-18T03:06:03.493 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2024-04-18T03:06:03.511 INFO:teuthology.lock.ops:Start node 'smithi136.front.sepia.ceph.com' reimaging 2024-04-18T03:06:03.511 INFO:teuthology.lock.ops:Updating [smithi136.front.sepia.ceph.com]: reset os type and version on server 2024-04-18T03:06:03.511 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-04-18T03:06:03.527 INFO:teuthology.lock.ops:Start node 'smithi162.front.sepia.ceph.com' reimaging 2024-04-18T03:06:03.527 INFO:teuthology.lock.ops:Updating [smithi162.front.sepia.ceph.com]: reset os type and version on server 2024-04-18T03:06:03.528 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-04-18T03:06:03.643 DEBUG:teuthology.provision.fog.smithi084:Requesting image smithi_rhel_8.6 (ID 5) 2024-04-18T03:06:03.655 DEBUG:teuthology.provision.fog.smithi136:Requesting image smithi_rhel_8.6 (ID 5) 2024-04-18T03:06:03.679 DEBUG:teuthology.provision.fog.smithi162:Requesting image smithi_rhel_8.6 (ID 5) 2024-04-18T03:06:03.686 INFO:teuthology.provision.fog.smithi084:Scheduling deploy of rhel 8.6 2024-04-18T03:06:03.704 INFO:teuthology.provision.fog.smithi136:Scheduling deploy of rhel 8.6 2024-04-18T03:06:03.730 INFO:teuthology.provision.fog.smithi162:Scheduling deploy of rhel 8.6 2024-04-18T03:06:04.034 INFO:teuthology.orchestra.console.smithi136:Power off 2024-04-18T03:06:04.034 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-18T03:06:04.047 INFO:teuthology.orchestra.console.smithi084:Power off 2024-04-18T03:06:04.048 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-18T03:06:04.059 INFO:teuthology.orchestra.console.smithi162:Power off 2024-04-18T03:06:04.059 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-18T03:06:04.067 DEBUG:teuthology.orchestra.console.smithi136:power off output: Chassis Power Control: Down/Off 2024-04-18T03:06:04.067 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:04.075 DEBUG:teuthology.orchestra.console.smithi084:power off output: Chassis Power Control: Down/Off 2024-04-18T03:06:04.075 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:04.083 DEBUG:teuthology.orchestra.console.smithi162:power off output: Chassis Power Control: Down/Off 2024-04-18T03:06:04.083 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:04.091 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-04-18T03:06:04.096 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-04-18T03:06:04.106 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-04-18T03:06:08.093 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:08.101 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:08.108 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:08.203 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-04-18T03:06:08.210 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-04-18T03:06:08.216 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-04-18T03:06:12.208 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:12.215 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:12.222 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:12.317 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-04-18T03:06:12.324 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off 2024-04-18T03:06:12.332 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is off 2024-04-18T03:06:12.425 INFO:teuthology.orchestra.console.smithi084:Power off completed 2024-04-18T03:06:12.433 INFO:teuthology.orchestra.console.smithi162:Power off completed 2024-04-18T03:06:12.526 INFO:teuthology.orchestra.console.smithi084:Power on 2024-04-18T03:06:12.526 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-18T03:06:12.534 INFO:teuthology.orchestra.console.smithi162:Power on 2024-04-18T03:06:12.534 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-18T03:06:12.548 DEBUG:teuthology.orchestra.console.smithi084:power on output: Chassis Power Control: Up/On 2024-04-18T03:06:12.548 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:12.556 DEBUG:teuthology.orchestra.console.smithi162:power on output: Chassis Power Control: Up/On 2024-04-18T03:06:12.556 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:12.568 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off 2024-04-18T03:06:12.576 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is off 2024-04-18T03:06:16.322 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:16.431 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off 2024-04-18T03:06:16.531 INFO:teuthology.orchestra.console.smithi136:Power off completed 2024-04-18T03:06:16.569 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:16.577 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:16.633 INFO:teuthology.orchestra.console.smithi136:Power on 2024-04-18T03:06:16.633 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-18T03:06:16.654 DEBUG:teuthology.orchestra.console.smithi136:power on output: Chassis Power Control: Up/On 2024-04-18T03:06:16.654 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:16.675 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off 2024-04-18T03:06:16.677 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off 2024-04-18T03:06:16.685 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is off 2024-04-18T03:06:20.677 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:20.683 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:20.690 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:20.786 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off 2024-04-18T03:06:20.792 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-04-18T03:06:20.798 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-04-18T03:06:20.893 INFO:teuthology.orchestra.console.smithi084:Power on completed 2024-04-18T03:06:20.899 INFO:teuthology.orchestra.console.smithi162:Power on completed 2024-04-18T03:06:20.994 INFO:teuthology.provision.fog.smithi084:Waiting for deploy to finish 2024-04-18T03:06:21.000 INFO:teuthology.provision.fog.smithi162:Waiting for deploy to finish 2024-04-18T03:06:24.790 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T03:06:24.900 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-04-18T03:06:25.000 INFO:teuthology.orchestra.console.smithi136:Power on completed 2024-04-18T03:06:25.101 INFO:teuthology.provision.fog.smithi136:Waiting for deploy to finish 2024-04-18T03:09:06.856 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:09:06.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:09:10.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:09:25.411 WARNING:teuthology.provision.fog.smithi084:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-04-18T03:09:25.411 WARNING:teuthology.provision.fog.smithi162:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-04-18T03:09:29.246 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-04-18T03:09:31.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:09:31.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:09:34.494 WARNING:teuthology.provision.fog.smithi162:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-04-18T03:09:34.498 WARNING:teuthology.provision.fog.smithi084:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-04-18T03:09:35.249 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:09:38.302 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-04-18T03:09:40.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:09:40.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:09:43.550 WARNING:teuthology.provision.fog.smithi162:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-04-18T03:09:43.551 WARNING:teuthology.provision.fog.smithi084:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-04-18T03:09:44.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:09:47.358 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-04-18T03:09:49.554 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:09:49.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:09:52.606 WARNING:teuthology.provision.fog.smithi084:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-04-18T03:09:52.610 WARNING:teuthology.provision.fog.smithi162:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-04-18T03:09:53.360 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:09:56.414 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-04-18T03:09:58.610 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:09:58.612 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:10:01.662 WARNING:teuthology.provision.fog.smithi162:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-04-18T03:10:02.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:10:07.669 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:10:58.664 WARNING:teuthology.provision.fog.smithi084:timed out 2024-04-18T03:11:02.423 WARNING:teuthology.provision.fog.smithi136:timed out 2024-04-18T03:11:04.667 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:11:04.893 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-04-18T03:11:04.894 DEBUG:teuthology.orchestra.run.smithi084:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-04-18T03:11:05.175 INFO:teuthology.provision.fog.smithi084:Node is ready 2024-04-18T03:11:05.175 DEBUG:teuthology.orchestra.run.smithi084:> hostname 2024-04-18T03:11:05.193 INFO:teuthology.orchestra.run.smithi084.stdout:smithi084.front.sepia.ceph.com 2024-04-18T03:11:05.193 DEBUG:teuthology.orchestra.run.smithi084:> grep smithi084.front.sepia.ceph.com /etc/hosts 2024-04-18T03:11:05.251 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84 smithi084.front.sepia.ceph.com smithi084 2024-04-18T03:11:05.251 DEBUG:teuthology.orchestra.run.smithi084:> sudo hostname smithi084 2024-04-18T03:11:05.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i -e 's/smithi084.front.sepia.ceph.com/smithi084/g' /etc/hosts 2024-04-18T03:11:05.400 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i -e 's/172.21.15.84/172.21.15.84/g' /etc/hosts 2024-04-18T03:11:05.468 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i -e 's/smithi084.front.sepia.ceph.com/smithi084/g' /etc/hostname 2024-04-18T03:11:05.536 DEBUG:teuthology.orchestra.run.smithi084:> sudo hostname smithi084 2024-04-18T03:11:05.604 DEBUG:teuthology.orchestra.run.smithi084:> cat /etc/os-release 2024-04-18T03:11:05.660 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="Red Hat Enterprise Linux" 2024-04-18T03:11:05.660 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="8.6 (Ootpa)" 2024-04-18T03:11:05.661 INFO:teuthology.orchestra.run.smithi084.stdout:ID="rhel" 2024-04-18T03:11:05.661 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="fedora" 2024-04-18T03:11:05.661 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="8.6" 2024-04-18T03:11:05.661 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el8" 2024-04-18T03:11:05.661 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-18T03:11:05.661 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2024-04-18T03:11:05.661 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-18T03:11:05.661 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://www.redhat.com/" 2024-04-18T03:11:05.661 INFO:teuthology.orchestra.run.smithi084.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-18T03:11:05.661 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-18T03:11:05.661 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-18T03:11:05.661 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-18T03:11:05.661 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-18T03:11:05.661 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-18T03:11:05.662 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-18T03:11:05.662 INFO:teuthology.provision.fog.smithi084:Deploy complete! 2024-04-18T03:11:05.664 DEBUG:teuthology.parallel:result is None 2024-04-18T03:11:07.673 WARNING:teuthology.provision.fog.smithi162:timed out 2024-04-18T03:11:08.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:11:08.671 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-04-18T03:11:08.671 DEBUG:teuthology.orchestra.run.smithi136:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-04-18T03:11:08.964 INFO:teuthology.provision.fog.smithi136:Node is ready 2024-04-18T03:11:08.964 DEBUG:teuthology.orchestra.run.smithi136:> hostname 2024-04-18T03:11:08.982 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136 2024-04-18T03:11:08.982 DEBUG:teuthology.orchestra.run.smithi136:> grep smithi136 /etc/hosts 2024-04-18T03:11:09.040 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136 smithi136.front.sepia.ceph.com smithi136 2024-04-18T03:11:09.041 DEBUG:teuthology.orchestra.run.smithi136:> sudo hostname smithi136 2024-04-18T03:11:09.164 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i -e 's/smithi136/smithi136/g' /etc/hosts 2024-04-18T03:11:09.193 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i -e 's/172.21.15.136/172.21.15.136/g' /etc/hosts 2024-04-18T03:11:09.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i -e 's/smithi136/smithi136/g' /etc/hostname 2024-04-18T03:11:09.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo hostname smithi136 2024-04-18T03:11:09.394 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-04-18T03:11:09.450 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="Red Hat Enterprise Linux" 2024-04-18T03:11:09.450 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="8.6 (Ootpa)" 2024-04-18T03:11:09.450 INFO:teuthology.orchestra.run.smithi136.stdout:ID="rhel" 2024-04-18T03:11:09.450 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="fedora" 2024-04-18T03:11:09.450 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="8.6" 2024-04-18T03:11:09.451 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el8" 2024-04-18T03:11:09.451 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-18T03:11:09.451 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2024-04-18T03:11:09.451 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-18T03:11:09.451 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://www.redhat.com/" 2024-04-18T03:11:09.451 INFO:teuthology.orchestra.run.smithi136.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-18T03:11:09.451 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-18T03:11:09.451 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-18T03:11:09.451 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-18T03:11:09.451 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-18T03:11:09.451 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-18T03:11:09.451 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-18T03:11:09.452 INFO:teuthology.provision.fog.smithi136:Deploy complete! 2024-04-18T03:11:09.452 DEBUG:teuthology.parallel:result is None 2024-04-18T03:11:13.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:11:13.903 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-04-18T03:11:13.904 DEBUG:teuthology.orchestra.run.smithi162:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-04-18T03:11:14.247 INFO:teuthology.provision.fog.smithi162:Node is ready 2024-04-18T03:11:14.247 DEBUG:teuthology.orchestra.run.smithi162:> hostname 2024-04-18T03:11:14.266 INFO:teuthology.orchestra.run.smithi162.stdout:smithi162 2024-04-18T03:11:14.266 DEBUG:teuthology.orchestra.run.smithi162:> grep smithi162 /etc/hosts 2024-04-18T03:11:14.326 INFO:teuthology.orchestra.run.smithi162.stdout:172.21.15.162 smithi162.front.sepia.ceph.com smithi162 2024-04-18T03:11:14.326 DEBUG:teuthology.orchestra.run.smithi162:> sudo hostname smithi162 2024-04-18T03:11:14.451 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i -e 's/smithi162/smithi162/g' /etc/hosts 2024-04-18T03:11:14.480 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i -e 's/172.21.15.162/172.21.15.162/g' /etc/hosts 2024-04-18T03:11:14.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i -e 's/smithi162/smithi162/g' /etc/hostname 2024-04-18T03:11:14.618 DEBUG:teuthology.orchestra.run.smithi162:> sudo hostname smithi162 2024-04-18T03:11:14.685 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-04-18T03:11:14.746 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="Red Hat Enterprise Linux" 2024-04-18T03:11:14.746 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="8.6 (Ootpa)" 2024-04-18T03:11:14.746 INFO:teuthology.orchestra.run.smithi162.stdout:ID="rhel" 2024-04-18T03:11:14.747 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="fedora" 2024-04-18T03:11:14.747 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="8.6" 2024-04-18T03:11:14.747 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el8" 2024-04-18T03:11:14.747 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-18T03:11:14.747 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2024-04-18T03:11:14.747 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-18T03:11:14.748 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://www.redhat.com/" 2024-04-18T03:11:14.748 INFO:teuthology.orchestra.run.smithi162.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-18T03:11:14.748 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-18T03:11:14.748 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-18T03:11:14.748 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-18T03:11:14.748 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-18T03:11:14.748 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-18T03:11:14.748 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-18T03:11:14.749 INFO:teuthology.provision.fog.smithi162:Deploy complete! 2024-04-18T03:11:14.750 DEBUG:teuthology.parallel:result is None 2024-04-18T03:11:15.060 INFO:teuthology.lock.ops:Checking smithi084.front.sepia.ceph.com 2024-04-18T03:11:15.060 INFO:teuthology.lock.ops:New key found. Updating... 2024-04-18T03:11:15.094 INFO:teuthology.lock.ops:Checking smithi136.front.sepia.ceph.com 2024-04-18T03:11:15.094 INFO:teuthology.lock.ops:New key found. Updating... 2024-04-18T03:11:15.119 INFO:teuthology.lock.ops:Checking smithi162.front.sepia.ceph.com 2024-04-18T03:11:15.119 INFO:teuthology.lock.ops:New key found. Updating... 2024-04-18T03:11:15.147 INFO:teuthology.lock.ops:Updating [smithi084.front.sepia.ceph.com]: set os type and version on server 2024-04-18T03:11:15.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-04-18T03:11:15.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:11:15.227 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-04-18T03:11:15.302 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-04-18T03:11:15.303 DEBUG:teuthology.orchestra.run.smithi084:> uname -m 2024-04-18T03:11:15.366 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2024-04-18T03:11:15.366 DEBUG:teuthology.orchestra.run.smithi084:> cat /etc/os-release 2024-04-18T03:11:15.427 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="Red Hat Enterprise Linux" 2024-04-18T03:11:15.427 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="8.6 (Ootpa)" 2024-04-18T03:11:15.427 INFO:teuthology.orchestra.run.smithi084.stdout:ID="rhel" 2024-04-18T03:11:15.428 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="fedora" 2024-04-18T03:11:15.428 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="8.6" 2024-04-18T03:11:15.428 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el8" 2024-04-18T03:11:15.428 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-18T03:11:15.428 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2024-04-18T03:11:15.428 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-18T03:11:15.428 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://www.redhat.com/" 2024-04-18T03:11:15.428 INFO:teuthology.orchestra.run.smithi084.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-18T03:11:15.428 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-18T03:11:15.428 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-04-18T03:11:15.428 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-18T03:11:15.428 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-18T03:11:15.428 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-18T03:11:15.428 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-18T03:11:15.429 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2024-04-18T03:11:15.453 INFO:teuthology.lock.ops:Updating [smithi136.front.sepia.ceph.com]: set os type and version on server 2024-04-18T03:11:15.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-04-18T03:11:15.453 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:11:15.528 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-04-18T03:11:15.598 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-04-18T03:11:15.598 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-04-18T03:11:15.658 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-04-18T03:11:15.658 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-04-18T03:11:15.719 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="Red Hat Enterprise Linux" 2024-04-18T03:11:15.719 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="8.6 (Ootpa)" 2024-04-18T03:11:15.719 INFO:teuthology.orchestra.run.smithi136.stdout:ID="rhel" 2024-04-18T03:11:15.719 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="fedora" 2024-04-18T03:11:15.720 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="8.6" 2024-04-18T03:11:15.720 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el8" 2024-04-18T03:11:15.720 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-18T03:11:15.720 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2024-04-18T03:11:15.720 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-18T03:11:15.720 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://www.redhat.com/" 2024-04-18T03:11:15.720 INFO:teuthology.orchestra.run.smithi136.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-18T03:11:15.720 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-18T03:11:15.720 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-18T03:11:15.720 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-18T03:11:15.720 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-18T03:11:15.720 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-18T03:11:15.720 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-18T03:11:15.721 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-04-18T03:11:15.744 INFO:teuthology.lock.ops:Updating [smithi162.front.sepia.ceph.com]: set os type and version on server 2024-04-18T03:11:15.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-04-18T03:11:15.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T03:11:15.815 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-04-18T03:11:15.887 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-04-18T03:11:15.887 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-04-18T03:11:15.947 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-04-18T03:11:15.947 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-04-18T03:11:16.007 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="Red Hat Enterprise Linux" 2024-04-18T03:11:16.007 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="8.6 (Ootpa)" 2024-04-18T03:11:16.007 INFO:teuthology.orchestra.run.smithi162.stdout:ID="rhel" 2024-04-18T03:11:16.007 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="fedora" 2024-04-18T03:11:16.007 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="8.6" 2024-04-18T03:11:16.007 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el8" 2024-04-18T03:11:16.007 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-18T03:11:16.007 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2024-04-18T03:11:16.007 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-18T03:11:16.008 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://www.redhat.com/" 2024-04-18T03:11:16.008 INFO:teuthology.orchestra.run.smithi162.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-18T03:11:16.008 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-18T03:11:16.008 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-04-18T03:11:16.008 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-18T03:11:16.008 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-18T03:11:16.008 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-18T03:11:16.008 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-18T03:11:16.008 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-04-18T03:11:16.027 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-18T03:11:16.129 INFO:teuthology.dispatcher.supervisor:Running job 7661419 2024-04-18T03:11:16.130 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-04-17_22:24:03-fs-reef-distro-default-smithi/7661419 --name teuthology-2024-04-17_22:24:03-fs-reef-distro-default-smithi --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} -- /home/teuthworker/archive/teuthology-2024-04-17_22:24:03-fs-reef-distro-default-smithi/7661419/orig.config.yaml 2024-04-18T03:11:16.133 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2024-04-17_22:24:03-fs-reef-distro-default-smithi/7661419 2024-04-18T03:11:16.133 INFO:teuthology.dispatcher.supervisor:Job PID: 825342 2024-04-18T03:11:16.134 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2024-04-18T03:13:16.214 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-18T03:15:16.388 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-18T03:17:16.555 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-18T03:19:16.731 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-18T03:21:16.869 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-18T03:23:17.040 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-18T03:25:17.173 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-18T03:25:17.241 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1