2024-03-07T12:02:32.220 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-07T12:02:32.220 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T12:02:32.401 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc752005c10>, signals=[15]) 2024-03-07T12:02:32.401 INFO:teuthology.lock.ops:Start node 'smithi131.front.sepia.ceph.com' reimaging 2024-03-07T12:02:32.402 INFO:teuthology.lock.ops:Updating [smithi131.front.sepia.ceph.com]: reset os type and version on server 2024-03-07T12:02:32.402 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2024-03-07T12:02:32.429 INFO:teuthology.lock.ops:Start node 'smithi142.front.sepia.ceph.com' reimaging 2024-03-07T12:02:32.430 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: reset os type and version on server 2024-03-07T12:02:32.430 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2024-03-07T12:02:32.446 INFO:teuthology.lock.ops:Start node 'smithi195.front.sepia.ceph.com' reimaging 2024-03-07T12:02:32.446 INFO:teuthology.lock.ops:Updating [smithi195.front.sepia.ceph.com]: reset os type and version on server 2024-03-07T12:02:32.446 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2024-03-07T12:02:32.539 DEBUG:teuthology.provision.fog.smithi131:Requesting image smithi_centos_9.stream (ID 10) 2024-03-07T12:02:32.557 DEBUG:teuthology.provision.fog.smithi142:Requesting image smithi_centos_9.stream (ID 10) 2024-03-07T12:02:32.578 DEBUG:teuthology.provision.fog.smithi195:Requesting image smithi_centos_9.stream (ID 10) 2024-03-07T12:02:32.593 INFO:teuthology.provision.fog.smithi142:Scheduling deploy of centos 9.stream 2024-03-07T12:02:32.598 INFO:teuthology.provision.fog.smithi131:Scheduling deploy of centos 9.stream 2024-03-07T12:02:32.627 INFO:teuthology.provision.fog.smithi195:Scheduling deploy of centos 9.stream 2024-03-07T12:02:32.828 INFO:teuthology.orchestra.console.smithi142:Power off 2024-03-07T12:02:32.828 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-07T12:02:32.851 DEBUG:teuthology.orchestra.console.smithi142:power off output: Chassis Power Control: Down/Off 2024-03-07T12:02:32.851 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T12:02:32.871 INFO:teuthology.orchestra.console.smithi131:Power off 2024-03-07T12:02:32.871 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-07T12:02:32.884 INFO:teuthology.orchestra.console.smithi195:Power off 2024-03-07T12:02:32.885 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-07T12:02:32.892 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is on 2024-03-07T12:02:32.892 DEBUG:teuthology.orchestra.console.smithi131:power off output: Chassis Power Control: Down/Off 2024-03-07T12:02:32.892 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T12:02:32.907 DEBUG:teuthology.orchestra.console.smithi195:power off output: Chassis Power Control: Down/Off 2024-03-07T12:02:32.907 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T12:02:32.915 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-03-07T12:02:32.929 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-03-07T12:02:36.896 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T12:02:36.916 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T12:02:36.930 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T12:02:37.006 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is on 2024-03-07T12:02:37.025 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-03-07T12:02:37.039 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-03-07T12:02:41.011 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T12:02:41.026 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T12:02:41.039 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T12:02:41.122 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is off 2024-03-07T12:02:41.136 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2024-03-07T12:02:41.148 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is off 2024-03-07T12:02:41.224 INFO:teuthology.orchestra.console.smithi142:Power off completed 2024-03-07T12:02:41.237 INFO:teuthology.orchestra.console.smithi131:Power off completed 2024-03-07T12:02:41.249 INFO:teuthology.orchestra.console.smithi195:Power off completed 2024-03-07T12:02:41.325 INFO:teuthology.orchestra.console.smithi142:Power on 2024-03-07T12:02:41.325 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-07T12:02:41.338 INFO:teuthology.orchestra.console.smithi131:Power on 2024-03-07T12:02:41.338 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-07T12:02:41.348 DEBUG:teuthology.orchestra.console.smithi142:power on output: Chassis Power Control: Up/On 2024-03-07T12:02:41.348 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T12:02:41.356 INFO:teuthology.orchestra.console.smithi195:Power on 2024-03-07T12:02:41.357 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-07T12:02:41.364 DEBUG:teuthology.orchestra.console.smithi131:power on output: Chassis Power Control: Up/On 2024-03-07T12:02:41.365 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T12:02:41.374 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is off 2024-03-07T12:02:41.377 DEBUG:teuthology.orchestra.console.smithi195:power on output: Chassis Power Control: Up/On 2024-03-07T12:02:41.377 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T12:02:41.389 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2024-03-07T12:02:41.401 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is off 2024-03-07T12:02:45.376 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T12:02:45.391 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T12:02:45.402 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T12:02:45.485 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is off 2024-03-07T12:02:45.501 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2024-03-07T12:02:45.510 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is off 2024-03-07T12:02:49.488 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T12:02:49.503 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T12:02:49.510 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T12:02:49.620 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-03-07T12:02:49.627 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is on 2024-03-07T12:02:49.676 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-03-07T12:02:49.721 INFO:teuthology.orchestra.console.smithi195:Power on completed 2024-03-07T12:02:49.728 INFO:teuthology.orchestra.console.smithi142:Power on completed 2024-03-07T12:02:49.777 INFO:teuthology.orchestra.console.smithi131:Power on completed 2024-03-07T12:02:49.822 INFO:teuthology.provision.fog.smithi195:Waiting for deploy to finish 2024-03-07T12:02:49.828 INFO:teuthology.provision.fog.smithi142:Waiting for deploy to finish 2024-03-07T12:02:49.878 INFO:teuthology.provision.fog.smithi131:Waiting for deploy to finish 2024-03-07T12:05:20.801 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:05:20.807 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:05:20.810 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:05:23.872 WARNING:teuthology.provision.fog.smithi142:[Errno None] Unable to connect to port 22 on 172.21.15.142 2024-03-07T12:05:23.872 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131 2024-03-07T12:05:23.873 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-03-07T12:05:29.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:05:29.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:05:29.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:05:32.956 WARNING:teuthology.provision.fog.smithi142:[Errno None] Unable to connect to port 22 on 172.21.15.142 2024-03-07T12:05:32.956 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131 2024-03-07T12:05:32.957 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-03-07T12:05:38.961 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:05:38.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:05:38.965 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:05:42.012 WARNING:teuthology.provision.fog.smithi142:[Errno None] Unable to connect to port 22 on 172.21.15.142 2024-03-07T12:05:42.044 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-03-07T12:05:42.044 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131 2024-03-07T12:05:48.016 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:05:48.045 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:05:48.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:05:51.072 WARNING:teuthology.provision.fog.smithi142:[Errno None] Unable to connect to port 22 on 172.21.15.142 2024-03-07T12:05:51.100 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131 2024-03-07T12:05:51.100 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-03-07T12:05:57.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:05:57.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:05:57.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:06:00.156 WARNING:teuthology.provision.fog.smithi142:[Errno None] Unable to connect to port 22 on 172.21.15.142 2024-03-07T12:06:00.157 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131 2024-03-07T12:06:00.157 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-03-07T12:06:06.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:06:06.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:06:06.167 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:06:09.244 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131 2024-03-07T12:06:09.244 WARNING:teuthology.provision.fog.smithi142:[Errno None] Unable to connect to port 22 on 172.21.15.142 2024-03-07T12:06:09.244 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-03-07T12:06:15.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:06:15.249 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:06:15.250 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:06:18.300 WARNING:teuthology.provision.fog.smithi131:[Errno None] Unable to connect to port 22 on 172.21.15.131 2024-03-07T12:06:24.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:07:15.271 WARNING:teuthology.provision.fog.smithi142:timed out 2024-03-07T12:07:15.272 WARNING:teuthology.provision.fog.smithi195:timed out 2024-03-07T12:07:21.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:07:21.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:07:21.523 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-07T12:07:21.524 DEBUG:teuthology.orchestra.run.smithi195:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-07T12:07:21.596 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-07T12:07:21.596 DEBUG:teuthology.orchestra.run.smithi142:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-07T12:07:21.839 INFO:teuthology.provision.fog.smithi195:Node is ready 2024-03-07T12:07:21.839 DEBUG:teuthology.orchestra.run.smithi195:> hostname 2024-03-07T12:07:21.854 INFO:teuthology.orchestra.run.smithi195.stdout:smithi195.front.sepia.ceph.com 2024-03-07T12:07:21.854 DEBUG:teuthology.orchestra.run.smithi195:> grep smithi195.front.sepia.ceph.com /etc/hosts 2024-03-07T12:07:21.908 INFO:teuthology.orchestra.run.smithi195.stdout:172.21.15.195 smithi195.front.sepia.ceph.com smithi195 2024-03-07T12:07:21.909 DEBUG:teuthology.orchestra.run.smithi195:> sudo hostname smithi195 2024-03-07T12:07:21.923 INFO:teuthology.provision.fog.smithi142:Node is ready 2024-03-07T12:07:21.923 DEBUG:teuthology.orchestra.run.smithi142:> hostname 2024-03-07T12:07:21.938 INFO:teuthology.orchestra.run.smithi142.stdout:smithi142.front.sepia.ceph.com 2024-03-07T12:07:21.939 DEBUG:teuthology.orchestra.run.smithi142:> grep smithi142.front.sepia.ceph.com /etc/hosts 2024-03-07T12:07:21.994 INFO:teuthology.orchestra.run.smithi142.stdout:172.21.15.142 smithi142.front.sepia.ceph.com smithi142 2024-03-07T12:07:21.994 DEBUG:teuthology.orchestra.run.smithi142:> sudo hostname smithi142 2024-03-07T12:07:22.006 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i -e 's/smithi195.front.sepia.ceph.com/smithi195/g' /etc/hosts 2024-03-07T12:07:22.034 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i -e 's/172.21.15.195/172.21.15.195/g' /etc/hosts 2024-03-07T12:07:22.100 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i -e 's/smithi195.front.sepia.ceph.com/smithi195/g' /etc/hostname 2024-03-07T12:07:22.101 DEBUG:teuthology.orchestra.run.smithi142:> sudo sed -i -e 's/smithi142.front.sepia.ceph.com/smithi142/g' /etc/hosts 2024-03-07T12:07:22.129 DEBUG:teuthology.orchestra.run.smithi142:> sudo sed -i -e 's/172.21.15.142/172.21.15.142/g' /etc/hosts 2024-03-07T12:07:22.167 DEBUG:teuthology.orchestra.run.smithi195:> sudo hostname smithi195 2024-03-07T12:07:22.196 DEBUG:teuthology.orchestra.run.smithi142:> sudo sed -i -e 's/smithi142.front.sepia.ceph.com/smithi142/g' /etc/hostname 2024-03-07T12:07:22.234 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2024-03-07T12:07:22.263 DEBUG:teuthology.orchestra.run.smithi142:> sudo hostname smithi142 2024-03-07T12:07:22.287 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream" 2024-03-07T12:07:22.288 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="9" 2024-03-07T12:07:22.288 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos" 2024-03-07T12:07:22.288 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora" 2024-03-07T12:07:22.288 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="9" 2024-03-07T12:07:22.288 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el9" 2024-03-07T12:07:22.288 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-07T12:07:22.288 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2024-03-07T12:07:22.288 INFO:teuthology.orchestra.run.smithi195.stdout:LOGO="fedora-logo-icon" 2024-03-07T12:07:22.288 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-07T12:07:22.288 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/" 2024-03-07T12:07:22.288 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-07T12:07:22.288 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-07T12:07:22.288 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-07T12:07:22.289 INFO:teuthology.provision.fog.smithi195:Deploy complete! 2024-03-07T12:07:22.291 DEBUG:teuthology.parallel:result is None 2024-03-07T12:07:22.329 DEBUG:teuthology.orchestra.run.smithi142:> cat /etc/os-release 2024-03-07T12:07:22.384 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="CentOS Stream" 2024-03-07T12:07:22.384 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="9" 2024-03-07T12:07:22.384 INFO:teuthology.orchestra.run.smithi142.stdout:ID="centos" 2024-03-07T12:07:22.384 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="rhel fedora" 2024-03-07T12:07:22.384 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="9" 2024-03-07T12:07:22.384 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el9" 2024-03-07T12:07:22.385 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-07T12:07:22.385 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31" 2024-03-07T12:07:22.385 INFO:teuthology.orchestra.run.smithi142.stdout:LOGO="fedora-logo-icon" 2024-03-07T12:07:22.385 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-07T12:07:22.385 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://centos.org/" 2024-03-07T12:07:22.385 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-07T12:07:22.385 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-07T12:07:22.385 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-07T12:07:22.385 INFO:teuthology.provision.fog.smithi142:Deploy complete! 2024-03-07T12:07:22.386 DEBUG:teuthology.parallel:result is None 2024-03-07T12:07:24.308 WARNING:teuthology.provision.fog.smithi131:timed out 2024-03-07T12:07:30.315 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:07:30.576 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-07T12:07:30.576 DEBUG:teuthology.orchestra.run.smithi131:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-07T12:07:30.882 INFO:teuthology.provision.fog.smithi131:Node is ready 2024-03-07T12:07:30.883 DEBUG:teuthology.orchestra.run.smithi131:> hostname 2024-03-07T12:07:30.897 INFO:teuthology.orchestra.run.smithi131.stdout:smithi131.front.sepia.ceph.com 2024-03-07T12:07:30.897 DEBUG:teuthology.orchestra.run.smithi131:> grep smithi131.front.sepia.ceph.com /etc/hosts 2024-03-07T12:07:30.953 INFO:teuthology.orchestra.run.smithi131.stdout:172.21.15.131 smithi131.front.sepia.ceph.com smithi131 2024-03-07T12:07:30.953 DEBUG:teuthology.orchestra.run.smithi131:> sudo hostname smithi131 2024-03-07T12:07:31.071 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i -e 's/smithi131.front.sepia.ceph.com/smithi131/g' /etc/hosts 2024-03-07T12:07:31.099 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i -e 's/172.21.15.131/172.21.15.131/g' /etc/hosts 2024-03-07T12:07:31.166 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i -e 's/smithi131.front.sepia.ceph.com/smithi131/g' /etc/hostname 2024-03-07T12:07:31.233 DEBUG:teuthology.orchestra.run.smithi131:> sudo hostname smithi131 2024-03-07T12:07:31.300 DEBUG:teuthology.orchestra.run.smithi131:> cat /etc/os-release 2024-03-07T12:07:31.354 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="CentOS Stream" 2024-03-07T12:07:31.354 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="9" 2024-03-07T12:07:31.354 INFO:teuthology.orchestra.run.smithi131.stdout:ID="centos" 2024-03-07T12:07:31.354 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE="rhel fedora" 2024-03-07T12:07:31.354 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="9" 2024-03-07T12:07:31.354 INFO:teuthology.orchestra.run.smithi131.stdout:PLATFORM_ID="platform:el9" 2024-03-07T12:07:31.354 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-07T12:07:31.354 INFO:teuthology.orchestra.run.smithi131.stdout:ANSI_COLOR="0;31" 2024-03-07T12:07:31.354 INFO:teuthology.orchestra.run.smithi131.stdout:LOGO="fedora-logo-icon" 2024-03-07T12:07:31.355 INFO:teuthology.orchestra.run.smithi131.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-07T12:07:31.355 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://centos.org/" 2024-03-07T12:07:31.355 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-07T12:07:31.355 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-07T12:07:31.355 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-07T12:07:31.355 INFO:teuthology.provision.fog.smithi131:Deploy complete! 2024-03-07T12:07:31.356 DEBUG:teuthology.parallel:result is None 2024-03-07T12:07:31.566 INFO:teuthology.lock.ops:Checking smithi131.front.sepia.ceph.com 2024-03-07T12:07:31.567 INFO:teuthology.lock.ops:New key found. Updating... 2024-03-07T12:07:31.592 INFO:teuthology.lock.ops:Checking smithi142.front.sepia.ceph.com 2024-03-07T12:07:31.592 INFO:teuthology.lock.ops:New key found. Updating... 2024-03-07T12:07:31.608 INFO:teuthology.lock.ops:Checking smithi195.front.sepia.ceph.com 2024-03-07T12:07:31.609 INFO:teuthology.lock.ops:New key found. Updating... 2024-03-07T12:07:31.635 INFO:teuthology.lock.ops:Updating [smithi131.front.sepia.ceph.com]: set os type and version on server 2024-03-07T12:07:31.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-03-07T12:07:31.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:07:31.726 DEBUG:teuthology.orchestra.run.smithi131:> true 2024-03-07T12:07:31.794 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-03-07T12:07:31.794 DEBUG:teuthology.orchestra.run.smithi131:> uname -m 2024-03-07T12:07:31.849 INFO:teuthology.orchestra.run.smithi131.stdout:x86_64 2024-03-07T12:07:31.850 DEBUG:teuthology.orchestra.run.smithi131:> cat /etc/os-release 2024-03-07T12:07:31.904 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="CentOS Stream" 2024-03-07T12:07:31.904 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="9" 2024-03-07T12:07:31.905 INFO:teuthology.orchestra.run.smithi131.stdout:ID="centos" 2024-03-07T12:07:31.905 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE="rhel fedora" 2024-03-07T12:07:31.905 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="9" 2024-03-07T12:07:31.905 INFO:teuthology.orchestra.run.smithi131.stdout:PLATFORM_ID="platform:el9" 2024-03-07T12:07:31.905 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-07T12:07:31.905 INFO:teuthology.orchestra.run.smithi131.stdout:ANSI_COLOR="0;31" 2024-03-07T12:07:31.905 INFO:teuthology.orchestra.run.smithi131.stdout:LOGO="fedora-logo-icon" 2024-03-07T12:07:31.905 INFO:teuthology.orchestra.run.smithi131.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-07T12:07:31.906 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://centos.org/" 2024-03-07T12:07:31.906 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-07T12:07:31.906 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-07T12:07:31.906 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-07T12:07:31.906 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2024-03-07T12:07:31.925 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: set os type and version on server 2024-03-07T12:07:31.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-03-07T12:07:31.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:07:32.015 DEBUG:teuthology.orchestra.run.smithi142:> true 2024-03-07T12:07:32.083 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-03-07T12:07:32.084 DEBUG:teuthology.orchestra.run.smithi142:> uname -m 2024-03-07T12:07:32.140 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2024-03-07T12:07:32.140 DEBUG:teuthology.orchestra.run.smithi142:> cat /etc/os-release 2024-03-07T12:07:32.197 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="CentOS Stream" 2024-03-07T12:07:32.197 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="9" 2024-03-07T12:07:32.197 INFO:teuthology.orchestra.run.smithi142.stdout:ID="centos" 2024-03-07T12:07:32.197 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="rhel fedora" 2024-03-07T12:07:32.197 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="9" 2024-03-07T12:07:32.197 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el9" 2024-03-07T12:07:32.197 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-07T12:07:32.197 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31" 2024-03-07T12:07:32.198 INFO:teuthology.orchestra.run.smithi142.stdout:LOGO="fedora-logo-icon" 2024-03-07T12:07:32.198 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-07T12:07:32.198 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://centos.org/" 2024-03-07T12:07:32.198 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-07T12:07:32.198 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-07T12:07:32.198 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-07T12:07:32.199 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2024-03-07T12:07:32.217 INFO:teuthology.lock.ops:Updating [smithi195.front.sepia.ceph.com]: set os type and version on server 2024-03-07T12:07:32.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-07T12:07:32.217 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T12:07:32.305 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-03-07T12:07:32.376 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-07T12:07:32.376 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2024-03-07T12:07:32.432 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2024-03-07T12:07:32.433 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2024-03-07T12:07:32.488 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream" 2024-03-07T12:07:32.489 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="9" 2024-03-07T12:07:32.489 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos" 2024-03-07T12:07:32.489 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora" 2024-03-07T12:07:32.489 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="9" 2024-03-07T12:07:32.489 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el9" 2024-03-07T12:07:32.490 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-07T12:07:32.490 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2024-03-07T12:07:32.490 INFO:teuthology.orchestra.run.smithi195.stdout:LOGO="fedora-logo-icon" 2024-03-07T12:07:32.490 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-07T12:07:32.490 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/" 2024-03-07T12:07:32.490 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-07T12:07:32.491 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-07T12:07:32.491 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-07T12:07:32.492 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2024-03-07T12:07:32.517 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T12:07:32.634 INFO:teuthology.dispatcher.supervisor:Running job 7584376 2024-03-07T12:07:32.634 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2024-03-07_10:36:38-fs-wip-vshankar-testing-20240307.013758-testing-default-smithi/7584376 --name vshankar-2024-03-07_10:36:38-fs-wip-vshankar-testing-20240307.013758-testing-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} -- /home/teuthworker/archive/vshankar-2024-03-07_10:36:38-fs-wip-vshankar-testing-20240307.013758-testing-default-smithi/7584376/orig.config.yaml 2024-03-07T12:07:32.638 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2024-03-07_10:36:38-fs-wip-vshankar-testing-20240307.013758-testing-default-smithi/7584376 2024-03-07T12:07:32.638 INFO:teuthology.dispatcher.supervisor:Job PID: 220212 2024-03-07T12:07:32.638 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2024-03-07T12:09:32.692 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T12:11:32.869 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T12:13:33.049 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T12:15:33.221 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T12:17:33.397 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T12:19:33.577 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T12:21:33.741 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T12:23:33.913 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T12:25:34.080 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T12:27:34.249 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T12:29:34.435 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T12:31:34.604 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T12:33:34.773 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T12:35:34.941 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T12:37:35.106 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T12:37:35.177 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1 2024-03-07T12:37:35.360 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi131 but it was locked by another job: /home/teuthworker/archive/vshankar-2024-03-07_10:36:38-fs-wip-vshankar-testing-20240307.013758-testing-default-smithi/7584422 2024-03-07T12:37:35.360 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi142 but it was locked by another job: /home/teuthworker/archive/vshankar-2024-03-07_10:36:38-fs-wip-vshankar-testing-20240307.013758-testing-default-smithi/7584421 2024-03-07T12:37:35.360 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi195 but it was locked by another job: /home/teuthworker/archive/vshankar-2024-03-07_10:36:38-fs-wip-vshankar-testing-20240307.013758-testing-default-smithi/7584422