2023-10-21T11:33:52.960 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-21T11:33:52.961 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T11:33:53.111 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff640334670>, signals=[15]) 2023-10-21T11:33:53.112 INFO:teuthology.lock.ops:Start node 'smithi031.front.sepia.ceph.com' reimaging 2023-10-21T11:33:53.113 INFO:teuthology.lock.ops:Updating [smithi031.front.sepia.ceph.com]: reset os type and version on server 2023-10-21T11:33:53.113 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2023-10-21T11:33:53.140 INFO:teuthology.lock.ops:Start node 'smithi125.front.sepia.ceph.com' reimaging 2023-10-21T11:33:53.140 INFO:teuthology.lock.ops:Updating [smithi125.front.sepia.ceph.com]: reset os type and version on server 2023-10-21T11:33:53.141 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server 2023-10-21T11:33:53.142 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /host {"name": "smithi031"} 2023-10-21T11:33:53.165 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /host {"name": "smithi125"} 2023-10-21T11:33:53.228 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /image {"name": "smithi_centos_9.stream"} 2023-10-21T11:33:53.231 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /image {"name": "smithi_centos_9.stream"} 2023-10-21T11:33:53.282 DEBUG:teuthology.provision.fog.smithi031:Requesting image smithi_centos_9.stream (ID 10) 2023-10-21T11:33:53.282 DEBUG:teuthology.provision.fog.smithi031:FOG request: PUT /host/136 {"imageID": 10} 2023-10-21T11:33:53.286 DEBUG:teuthology.provision.fog.smithi125:Requesting image smithi_centos_9.stream (ID 10) 2023-10-21T11:33:53.286 DEBUG:teuthology.provision.fog.smithi125:FOG request: PUT /host/230 {"imageID": 10} 2023-10-21T11:33:53.327 INFO:teuthology.provision.fog.smithi031:Scheduling deploy of centos 9.stream 2023-10-21T11:33:53.327 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None 2023-10-21T11:33:53.344 INFO:teuthology.provision.fog.smithi125:Scheduling deploy of centos 9.stream 2023-10-21T11:33:53.345 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-10-21T11:33:53.406 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /tasktype {"name": "deploy"} 2023-10-21T11:33:53.421 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /tasktype {"name": "deploy"} 2023-10-21T11:33:53.442 DEBUG:teuthology.provision.fog.smithi031:FOG request: POST /host/136/task {"taskTypeID": 1} 2023-10-21T11:33:53.458 DEBUG:teuthology.provision.fog.smithi125:FOG request: POST /host/230/task {"taskTypeID": 1} 2023-10-21T11:33:53.506 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None 2023-10-21T11:33:53.524 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-10-21T11:33:53.628 INFO:teuthology.orchestra.console.smithi031:Power off 2023-10-21T11:33:53.629 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-21T11:33:53.643 INFO:teuthology.orchestra.console.smithi125:Power off 2023-10-21T11:33:53.644 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-21T11:33:53.652 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2023-10-21T11:33:53.652 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T11:33:53.664 DEBUG:teuthology.orchestra.console.smithi125:power off output: Chassis Power Control: Down/Off 2023-10-21T11:33:53.664 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T11:33:53.673 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2023-10-21T11:33:53.684 DEBUG:teuthology.orchestra.console.smithi125:check power output: Chassis Power is off 2023-10-21T11:33:53.775 INFO:teuthology.orchestra.console.smithi031:Power off completed 2023-10-21T11:33:53.786 INFO:teuthology.orchestra.console.smithi125:Power off completed 2023-10-21T11:33:53.876 INFO:teuthology.orchestra.console.smithi031:Power on 2023-10-21T11:33:53.877 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-10-21T11:33:53.886 INFO:teuthology.orchestra.console.smithi125:Power on 2023-10-21T11:33:53.888 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-10-21T11:33:53.898 DEBUG:teuthology.orchestra.console.smithi031:power on output: Chassis Power Control: Up/On 2023-10-21T11:33:53.899 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T11:33:53.909 DEBUG:teuthology.orchestra.console.smithi125:power on output: Chassis Power Control: Up/On 2023-10-21T11:33:53.909 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T11:33:53.920 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2023-10-21T11:33:53.931 DEBUG:teuthology.orchestra.console.smithi125:check power output: Chassis Power is off 2023-10-21T11:33:57.922 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T11:33:57.932 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T11:33:58.034 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2023-10-21T11:33:58.042 DEBUG:teuthology.orchestra.console.smithi125:check power output: Chassis Power is off 2023-10-21T11:34:02.038 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T11:34:02.045 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T11:34:02.147 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-10-21T11:34:02.153 DEBUG:teuthology.orchestra.console.smithi125:check power output: Chassis Power is on 2023-10-21T11:34:02.249 INFO:teuthology.orchestra.console.smithi031:Power on completed 2023-10-21T11:34:02.255 INFO:teuthology.orchestra.console.smithi125:Power on completed 2023-10-21T11:34:02.351 INFO:teuthology.provision.fog.smithi031:Waiting for deploy to finish 2023-10-21T11:34:02.351 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None 2023-10-21T11:34:02.355 INFO:teuthology.provision.fog.smithi125:Waiting for deploy to finish 2023-10-21T11:34:02.356 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-10-21T11:34:17.442 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None 2023-10-21T11:34:17.444 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-10-21T11:34:32.530 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None 2023-10-21T11:34:32.532 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-10-21T11:34:47.615 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-10-21T11:34:47.617 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None 2023-10-21T11:35:02.703 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-10-21T11:35:02.705 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None 2023-10-21T11:35:17.782 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-10-21T11:35:17.784 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None 2023-10-21T11:35:32.860 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-10-21T11:35:32.861 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None 2023-10-21T11:35:47.934 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None 2023-10-21T11:35:47.936 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-10-21T11:36:03.014 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-10-21T11:36:03.015 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None 2023-10-21T11:36:18.086 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-10-21T11:36:18.088 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None 2023-10-21T11:36:18.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T11:36:33.162 DEBUG:teuthology.provision.fog.smithi125:FOG request: GET /task/active None 2023-10-21T11:36:33.218 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T11:36:36.478 WARNING:teuthology.provision.fog.smithi031:[Errno None] Unable to connect to port 22 on 172.21.15.31 2023-10-21T11:36:42.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T11:36:45.566 WARNING:teuthology.provision.fog.smithi031:[Errno None] Unable to connect to port 22 on 172.21.15.31 2023-10-21T11:36:51.570 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T11:36:51.578 WARNING:teuthology.provision.fog.smithi125:[Errno None] Unable to connect to port 22 on 172.21.15.125 2023-10-21T11:36:54.654 WARNING:teuthology.provision.fog.smithi031:[Errno None] Unable to connect to port 22 on 172.21.15.31 2023-10-21T11:36:57.582 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T11:37:00.638 WARNING:teuthology.provision.fog.smithi125:[Errno None] Unable to connect to port 22 on 172.21.15.125 2023-10-21T11:37:00.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T11:37:03.706 WARNING:teuthology.provision.fog.smithi031:[Errno None] Unable to connect to port 22 on 172.21.15.31 2023-10-21T11:37:06.642 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T11:37:09.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T11:37:09.726 WARNING:teuthology.provision.fog.smithi125:[Errno None] Unable to connect to port 22 on 172.21.15.125 2023-10-21T11:37:15.731 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T11:37:18.818 WARNING:teuthology.provision.fog.smithi125:[Errno None] Unable to connect to port 22 on 172.21.15.125 2023-10-21T11:37:19.994 WARNING:teuthology.provision.fog.smithi031:[Errno None] Unable to connect to port 22 on 172.21.15.31 2023-10-21T11:37:24.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T11:37:25.997 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T11:37:27.898 WARNING:teuthology.provision.fog.smithi125:[Errno None] Unable to connect to port 22 on 172.21.15.125 2023-10-21T11:37:29.050 WARNING:teuthology.provision.fog.smithi031:[Errno None] Unable to connect to port 22 on 172.21.15.31 2023-10-21T11:37:33.902 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T11:37:35.053 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T11:37:38.106 WARNING:teuthology.provision.fog.smithi031:[Errno None] Unable to connect to port 22 on 172.21.15.31 2023-10-21T11:37:44.111 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T11:37:44.394 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-10-21T11:37:44.395 DEBUG:teuthology.orchestra.run.smithi031:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-10-21T11:38:00.668 INFO:teuthology.provision.fog.smithi031:Node is ready 2023-10-21T11:38:00.669 DEBUG:teuthology.orchestra.run.smithi031:> hostname 2023-10-21T11:38:00.688 INFO:teuthology.orchestra.run.smithi031.stdout:smithi046 2023-10-21T11:38:00.688 DEBUG:teuthology.orchestra.run.smithi031:> grep smithi046 /etc/hosts 2023-10-21T11:38:00.746 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.46 smithi046.front.sepia.ceph.com smithi046 2023-10-21T11:38:00.746 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname smithi031 2023-10-21T11:38:00.886 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i -e 's/smithi046/smithi031/g' /etc/hosts 2023-10-21T11:38:00.921 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i -e 's/172.21.15.46/172.21.15.31/g' /etc/hosts 2023-10-21T11:38:00.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i -e 's/smithi046/smithi031/g' /etc/hostname 2023-10-21T11:38:01.069 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname smithi031 2023-10-21T11:38:01.140 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2023-10-21T11:38:01.196 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2023-10-21T11:38:01.197 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9" 2023-10-21T11:38:01.197 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2023-10-21T11:38:01.197 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2023-10-21T11:38:01.197 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9" 2023-10-21T11:38:01.198 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9" 2023-10-21T11:38:01.198 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-21T11:38:01.198 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2023-10-21T11:38:01.198 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon" 2023-10-21T11:38:01.198 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-21T11:38:01.199 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2023-10-21T11:38:01.199 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-21T11:38:01.199 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-21T11:38:01.199 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-21T11:38:01.200 INFO:teuthology.provision.fog.smithi031:Deploy complete! 2023-10-21T11:38:01.201 DEBUG:teuthology.parallel:result is None 2023-10-21T11:38:33.938 WARNING:teuthology.provision.fog.smithi125:timed out 2023-10-21T11:38:39.946 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T11:38:40.213 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-10-21T11:38:40.213 DEBUG:teuthology.orchestra.run.smithi125:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-10-21T11:38:40.534 INFO:teuthology.provision.fog.smithi125:Node is ready 2023-10-21T11:38:40.534 DEBUG:teuthology.orchestra.run.smithi125:> hostname 2023-10-21T11:38:40.550 INFO:teuthology.orchestra.run.smithi125.stdout:smithi125.front.sepia.ceph.com 2023-10-21T11:38:40.550 DEBUG:teuthology.orchestra.run.smithi125:> grep smithi125.front.sepia.ceph.com /etc/hosts 2023-10-21T11:38:40.605 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125 smithi125.front.sepia.ceph.com smithi125 2023-10-21T11:38:40.605 DEBUG:teuthology.orchestra.run.smithi125:> sudo hostname smithi125 2023-10-21T11:38:40.713 DEBUG:teuthology.orchestra.run.smithi125:> sudo sed -i -e 's/smithi125.front.sepia.ceph.com/smithi125/g' /etc/hosts 2023-10-21T11:38:40.742 DEBUG:teuthology.orchestra.run.smithi125:> sudo sed -i -e 's/172.21.15.125/172.21.15.125/g' /etc/hosts 2023-10-21T11:38:40.809 DEBUG:teuthology.orchestra.run.smithi125:> sudo sed -i -e 's/smithi125.front.sepia.ceph.com/smithi125/g' /etc/hostname 2023-10-21T11:38:40.877 DEBUG:teuthology.orchestra.run.smithi125:> sudo hostname smithi125 2023-10-21T11:38:40.943 DEBUG:teuthology.orchestra.run.smithi125:> cat /etc/os-release 2023-10-21T11:38:40.997 INFO:teuthology.orchestra.run.smithi125.stdout:NAME="CentOS Stream" 2023-10-21T11:38:40.997 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION="9" 2023-10-21T11:38:40.998 INFO:teuthology.orchestra.run.smithi125.stdout:ID="centos" 2023-10-21T11:38:40.998 INFO:teuthology.orchestra.run.smithi125.stdout:ID_LIKE="rhel fedora" 2023-10-21T11:38:40.998 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION_ID="9" 2023-10-21T11:38:40.998 INFO:teuthology.orchestra.run.smithi125.stdout:PLATFORM_ID="platform:el9" 2023-10-21T11:38:40.998 INFO:teuthology.orchestra.run.smithi125.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-21T11:38:40.999 INFO:teuthology.orchestra.run.smithi125.stdout:ANSI_COLOR="0;31" 2023-10-21T11:38:40.999 INFO:teuthology.orchestra.run.smithi125.stdout:LOGO="fedora-logo-icon" 2023-10-21T11:38:40.999 INFO:teuthology.orchestra.run.smithi125.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-21T11:38:40.999 INFO:teuthology.orchestra.run.smithi125.stdout:HOME_URL="https://centos.org/" 2023-10-21T11:38:40.999 INFO:teuthology.orchestra.run.smithi125.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-21T11:38:41.000 INFO:teuthology.orchestra.run.smithi125.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-21T11:38:41.000 INFO:teuthology.orchestra.run.smithi125.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-21T11:38:41.000 INFO:teuthology.provision.fog.smithi125:Deploy complete! 2023-10-21T11:38:41.001 DEBUG:teuthology.parallel:result is None 2023-10-21T11:38:41.174 INFO:teuthology.lock.ops:Checking smithi031.front.sepia.ceph.com 2023-10-21T11:38:41.175 INFO:teuthology.lock.ops:New key found. Updating... 2023-10-21T11:38:41.199 INFO:teuthology.lock.ops:Checking smithi125.front.sepia.ceph.com 2023-10-21T11:38:41.199 INFO:teuthology.lock.ops:New key found. Updating... 2023-10-21T11:38:41.217 INFO:teuthology.lock.ops:Updating [smithi031.front.sepia.ceph.com]: set os type and version on server 2023-10-21T11:38:41.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-21T11:38:41.218 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T11:38:41.316 DEBUG:teuthology.orchestra.run.smithi031:> true 2023-10-21T11:38:41.500 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-21T11:38:41.501 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2023-10-21T11:38:41.555 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2023-10-21T11:38:41.555 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2023-10-21T11:38:41.610 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2023-10-21T11:38:41.610 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9" 2023-10-21T11:38:41.611 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2023-10-21T11:38:41.611 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2023-10-21T11:38:41.611 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9" 2023-10-21T11:38:41.611 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9" 2023-10-21T11:38:41.612 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-21T11:38:41.612 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2023-10-21T11:38:41.612 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon" 2023-10-21T11:38:41.612 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-21T11:38:41.612 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2023-10-21T11:38:41.613 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-21T11:38:41.613 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-21T11:38:41.613 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-21T11:38:41.614 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2023-10-21T11:38:41.633 INFO:teuthology.lock.ops:Updating [smithi125.front.sepia.ceph.com]: set os type and version on server 2023-10-21T11:38:41.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com' 2023-10-21T11:38:41.634 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-21T11:38:41.717 DEBUG:teuthology.orchestra.run.smithi125:> true 2023-10-21T11:38:41.784 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi125.front.sepia.ceph.com' 2023-10-21T11:38:41.784 DEBUG:teuthology.orchestra.run.smithi125:> uname -m 2023-10-21T11:38:41.839 INFO:teuthology.orchestra.run.smithi125.stdout:x86_64 2023-10-21T11:38:41.839 DEBUG:teuthology.orchestra.run.smithi125:> cat /etc/os-release 2023-10-21T11:38:41.893 INFO:teuthology.orchestra.run.smithi125.stdout:NAME="CentOS Stream" 2023-10-21T11:38:41.893 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION="9" 2023-10-21T11:38:41.893 INFO:teuthology.orchestra.run.smithi125.stdout:ID="centos" 2023-10-21T11:38:41.894 INFO:teuthology.orchestra.run.smithi125.stdout:ID_LIKE="rhel fedora" 2023-10-21T11:38:41.894 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION_ID="9" 2023-10-21T11:38:41.894 INFO:teuthology.orchestra.run.smithi125.stdout:PLATFORM_ID="platform:el9" 2023-10-21T11:38:41.894 INFO:teuthology.orchestra.run.smithi125.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-21T11:38:41.895 INFO:teuthology.orchestra.run.smithi125.stdout:ANSI_COLOR="0;31" 2023-10-21T11:38:41.895 INFO:teuthology.orchestra.run.smithi125.stdout:LOGO="fedora-logo-icon" 2023-10-21T11:38:41.895 INFO:teuthology.orchestra.run.smithi125.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-21T11:38:41.895 INFO:teuthology.orchestra.run.smithi125.stdout:HOME_URL="https://centos.org/" 2023-10-21T11:38:41.895 INFO:teuthology.orchestra.run.smithi125.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-21T11:38:41.896 INFO:teuthology.orchestra.run.smithi125.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-21T11:38:41.896 INFO:teuthology.orchestra.run.smithi125.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-21T11:38:41.896 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server 2023-10-21T11:38:41.916 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T11:38:42.007 INFO:teuthology.dispatcher.supervisor:Running job 7434054 2023-10-21T11:38:42.007 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-10-20_18:40:51-rados-main-distro-default-smithi/7434054 --name yuriw-2023-10-20_18:40:51-rados-main-distro-default-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/rados_api_tests} -- /home/teuthworker/archive/yuriw-2023-10-20_18:40:51-rados-main-distro-default-smithi/7434054/orig.config.yaml 2023-10-21T11:38:42.011 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-10-20_18:40:51-rados-main-distro-default-smithi/7434054 2023-10-21T11:38:42.011 INFO:teuthology.dispatcher.supervisor:Job PID: 199583 2023-10-21T11:38:42.012 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-10-21T11:40:42.111 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T11:42:42.287 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T11:44:42.464 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T11:46:42.627 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T11:48:42.796 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T11:50:42.963 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T11:52:43.135 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T11:54:43.299 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T11:56:43.475 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T11:58:43.651 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T12:00:43.823 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T12:02:43.993 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-21T12:02:44.066 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120 2023-10-21T12:02:44.174 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2023-10-21T12:02:44.175 INFO:teuthology.nuke:Checking targets against current locks 2023-10-21T12:02:44.225 DEBUG:teuthology.nuke:shortname: smithi031 2023-10-21T12:02:44.226 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-21T12:02:44.248 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-20_18:40:51-rados-main-distro-default-smithi/7434054', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-21 11:33:52.118967', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDD49OA/6nLBne9JaJ/MANk3OJYM8kG1FgKV9PTYIRC4l7sy8pUa+1C4ERADKO70wsFuRO2V0cBVq16yLIo0qYQ='} 2023-10-21T12:02:44.251 DEBUG:teuthology.nuke:shortname: smithi125 2023-10-21T12:02:44.252 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-21T12:02:44.271 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi125.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-20_18:40:51-rados-main-distro-default-smithi/7434054', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-21 11:33:52.120600', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK3hna9eqZrXdj2agYxGniLsFHhQVtrg8CvYFtu/xpk29T+UYEKlDvOJUPDYrMspuawo5nRlv8fU7HMP1+61Cd0='} 2023-10-21T12:02:44.296 INFO:teuthology.orchestra.console.smithi031:Power off 2023-10-21T12:02:44.296 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-21T12:02:44.319 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2023-10-21T12:02:44.320 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T12:02:44.336 INFO:teuthology.orchestra.console.smithi125:Power off 2023-10-21T12:02:44.337 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-21T12:02:44.345 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2023-10-21T12:02:44.357 DEBUG:teuthology.orchestra.console.smithi125:power off output: Chassis Power Control: Down/Off 2023-10-21T12:02:44.357 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-21T12:02:44.379 DEBUG:teuthology.orchestra.console.smithi125:check power output: Chassis Power is off 2023-10-21T12:02:44.446 INFO:teuthology.orchestra.console.smithi031:Power off completed 2023-10-21T12:02:44.481 INFO:teuthology.orchestra.console.smithi125:Power off completed 2023-10-21T12:02:44.598 INFO:teuthology.lock.ops:unlocked: smithi031.front.sepia.ceph.com 2023-10-21T12:02:44.623 INFO:teuthology.lock.ops:unlocked: smithi125.front.sepia.ceph.com