2023-11-11T00:06:43.169 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-11T00:06:43.170 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-11T00:06:43.311 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f597eba3ee0>, signals=[15]) 2023-11-11T00:06:43.313 INFO:teuthology.lock.ops:Start node 'smithi152.front.sepia.ceph.com' reimaging 2023-11-11T00:06:43.313 INFO:teuthology.lock.ops:Updating [smithi152.front.sepia.ceph.com]: reset os type and version on server 2023-11-11T00:06:43.314 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2023-11-11T00:06:43.334 INFO:teuthology.lock.ops:Start node 'smithi169.front.sepia.ceph.com' reimaging 2023-11-11T00:06:43.334 INFO:teuthology.lock.ops:Updating [smithi169.front.sepia.ceph.com]: reset os type and version on server 2023-11-11T00:06:43.334 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2023-11-11T00:06:43.336 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /host {"name": "smithi152"} 2023-11-11T00:06:43.359 DEBUG:teuthology.provision.fog.smithi169:FOG request: GET /host {"name": "smithi169"} 2023-11-11T00:06:43.403 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /image {"name": "smithi_centos_9.stream"} 2023-11-11T00:06:43.431 DEBUG:teuthology.provision.fog.smithi169:FOG request: GET /image {"name": "smithi_centos_9.stream"} 2023-11-11T00:06:43.444 DEBUG:teuthology.provision.fog.smithi152:Requesting image smithi_centos_9.stream (ID 10) 2023-11-11T00:06:43.444 DEBUG:teuthology.provision.fog.smithi152:FOG request: PUT /host/257 {"imageID": 10} 2023-11-11T00:06:43.486 DEBUG:teuthology.provision.fog.smithi169:Requesting image smithi_centos_9.stream (ID 10) 2023-11-11T00:06:43.486 DEBUG:teuthology.provision.fog.smithi169:FOG request: PUT /host/274 {"imageID": 10} 2023-11-11T00:06:43.501 INFO:teuthology.provision.fog.smithi152:Scheduling deploy of centos 9.stream 2023-11-11T00:06:43.501 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-11-11T00:06:43.525 INFO:teuthology.provision.fog.smithi169:Scheduling deploy of centos 9.stream 2023-11-11T00:06:43.526 DEBUG:teuthology.provision.fog.smithi169:FOG request: GET /task/active None 2023-11-11T00:06:43.560 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /tasktype {"name": "deploy"} 2023-11-11T00:06:43.576 DEBUG:teuthology.provision.fog.smithi169:FOG request: GET /tasktype {"name": "deploy"} 2023-11-11T00:06:43.599 DEBUG:teuthology.provision.fog.smithi152:FOG request: POST /host/257/task {"taskTypeID": 1} 2023-11-11T00:06:43.606 DEBUG:teuthology.provision.fog.smithi169:FOG request: POST /host/274/task {"taskTypeID": 1} 2023-11-11T00:06:43.650 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-11-11T00:06:43.671 DEBUG:teuthology.provision.fog.smithi169:FOG request: GET /task/active None 2023-11-11T00:06:43.744 INFO:teuthology.orchestra.console.smithi152:Power off 2023-11-11T00:06:43.745 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-11T00:06:43.764 INFO:teuthology.orchestra.console.smithi169:Power off 2023-11-11T00:06:43.765 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-11T00:06:43.776 DEBUG:teuthology.orchestra.console.smithi152:power off output: Chassis Power Control: Down/Off 2023-11-11T00:06:43.777 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-11T00:06:43.796 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2023-11-11T00:06:43.797 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-11T00:06:43.813 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2023-11-11T00:06:43.828 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2023-11-11T00:06:47.819 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-11T00:06:47.829 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-11T00:06:47.928 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2023-11-11T00:06:47.939 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2023-11-11T00:06:51.933 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-11T00:06:51.940 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-11T00:06:52.044 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is off 2023-11-11T00:06:52.053 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2023-11-11T00:06:52.146 INFO:teuthology.orchestra.console.smithi152:Power off completed 2023-11-11T00:06:52.154 INFO:teuthology.orchestra.console.smithi169:Power off completed 2023-11-11T00:06:52.247 INFO:teuthology.orchestra.console.smithi152:Power on 2023-11-11T00:06:52.248 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-11-11T00:06:52.256 INFO:teuthology.orchestra.console.smithi169:Power on 2023-11-11T00:06:52.257 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-11-11T00:06:52.303 DEBUG:teuthology.orchestra.console.smithi152:power on output: Chassis Power Control: Up/On 2023-11-11T00:06:52.304 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-11T00:06:52.314 DEBUG:teuthology.orchestra.console.smithi169:power on output: Chassis Power Control: Up/On 2023-11-11T00:06:52.314 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-11T00:06:52.337 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is off 2023-11-11T00:06:52.355 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2023-11-11T00:06:56.342 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-11T00:06:56.356 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-11T00:06:56.453 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is off 2023-11-11T00:06:56.467 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2023-11-11T00:07:00.458 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-11T00:07:00.468 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-11T00:07:00.567 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2023-11-11T00:07:00.576 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2023-11-11T00:07:00.670 INFO:teuthology.orchestra.console.smithi152:Power on completed 2023-11-11T00:07:00.678 INFO:teuthology.orchestra.console.smithi169:Power on completed 2023-11-11T00:07:00.771 INFO:teuthology.provision.fog.smithi152:Waiting for deploy to finish 2023-11-11T00:07:00.772 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-11-11T00:07:00.778 INFO:teuthology.provision.fog.smithi169:Waiting for deploy to finish 2023-11-11T00:07:00.779 DEBUG:teuthology.provision.fog.smithi169:FOG request: GET /task/active None 2023-11-11T00:07:15.848 DEBUG:teuthology.provision.fog.smithi169:FOG request: GET /task/active None 2023-11-11T00:07:15.850 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-11-11T00:07:30.920 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-11-11T00:07:30.922 DEBUG:teuthology.provision.fog.smithi169:FOG request: GET /task/active None 2023-11-11T00:07:45.998 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-11-11T00:07:46.001 DEBUG:teuthology.provision.fog.smithi169:FOG request: GET /task/active None 2023-11-11T00:08:01.072 DEBUG:teuthology.provision.fog.smithi169:FOG request: GET /task/active None 2023-11-11T00:08:01.074 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-11-11T00:08:16.142 DEBUG:teuthology.provision.fog.smithi169:FOG request: GET /task/active None 2023-11-11T00:08:16.144 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-11-11T00:08:31.226 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-11-11T00:08:31.228 DEBUG:teuthology.provision.fog.smithi169:FOG request: GET /task/active None 2023-11-11T00:08:46.310 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-11-11T00:08:46.312 DEBUG:teuthology.provision.fog.smithi169:FOG request: GET /task/active None 2023-11-11T00:09:01.398 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-11-11T00:09:01.400 DEBUG:teuthology.provision.fog.smithi169:FOG request: GET /task/active None 2023-11-11T00:09:16.494 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-11-11T00:09:16.497 DEBUG:teuthology.provision.fog.smithi169:FOG request: GET /task/active None 2023-11-11T00:09:31.582 DEBUG:teuthology.provision.fog.smithi169:FOG request: GET /task/active None 2023-11-11T00:09:31.584 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-11-11T00:09:46.682 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-11-11T00:09:46.684 DEBUG:teuthology.provision.fog.smithi169:FOG request: GET /task/active None 2023-11-11T00:09:46.757 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:09:46.761 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:09:49.818 WARNING:teuthology.provision.fog.smithi169:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-11-11T00:09:49.819 WARNING:teuthology.provision.fog.smithi152:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-11-11T00:09:55.826 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:09:55.829 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:09:58.910 WARNING:teuthology.provision.fog.smithi152:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-11-11T00:09:58.910 WARNING:teuthology.provision.fog.smithi169:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-11-11T00:10:04.915 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:10:04.917 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:10:07.998 WARNING:teuthology.provision.fog.smithi152:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-11-11T00:10:07.999 WARNING:teuthology.provision.fog.smithi169:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-11-11T00:10:14.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:10:14.005 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:10:17.082 WARNING:teuthology.provision.fog.smithi152:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-11-11T00:10:17.083 WARNING:teuthology.provision.fog.smithi169:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-11-11T00:10:23.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:10:23.089 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:10:26.142 WARNING:teuthology.provision.fog.smithi152:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-11-11T00:10:26.175 WARNING:teuthology.provision.fog.smithi169:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-11-11T00:10:32.146 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:10:32.176 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:10:35.230 WARNING:teuthology.provision.fog.smithi152:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-11-11T00:10:35.262 WARNING:teuthology.provision.fog.smithi169:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-11-11T00:10:41.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:10:41.263 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:10:44.314 WARNING:teuthology.provision.fog.smithi152:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-11-11T00:10:44.315 WARNING:teuthology.provision.fog.smithi169:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-11-11T00:10:50.318 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:10:50.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:11:50.379 WARNING:teuthology.provision.fog.smithi152:timed out 2023-11-11T00:11:50.381 WARNING:teuthology.provision.fog.smithi169:timed out 2023-11-11T00:11:56.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:11:56.389 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:11:56.629 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-11-11T00:11:56.630 DEBUG:teuthology.orchestra.run.smithi152:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-11-11T00:11:56.642 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-11-11T00:11:56.642 DEBUG:teuthology.orchestra.run.smithi169:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-11-11T00:11:56.966 INFO:teuthology.provision.fog.smithi152:Node is ready 2023-11-11T00:11:56.966 DEBUG:teuthology.orchestra.run.smithi152:> hostname 2023-11-11T00:11:56.983 INFO:teuthology.provision.fog.smithi169:Node is ready 2023-11-11T00:11:56.983 DEBUG:teuthology.orchestra.run.smithi169:> hostname 2023-11-11T00:11:56.985 INFO:teuthology.orchestra.run.smithi152.stdout:smithi152.front.sepia.ceph.com 2023-11-11T00:11:56.985 DEBUG:teuthology.orchestra.run.smithi152:> grep smithi152.front.sepia.ceph.com /etc/hosts 2023-11-11T00:11:56.999 INFO:teuthology.orchestra.run.smithi169.stdout:smithi169.front.sepia.ceph.com 2023-11-11T00:11:57.000 DEBUG:teuthology.orchestra.run.smithi169:> grep smithi169.front.sepia.ceph.com /etc/hosts 2023-11-11T00:11:57.040 INFO:teuthology.orchestra.run.smithi152.stdout:172.21.15.152 smithi152.front.sepia.ceph.com smithi152 2023-11-11T00:11:57.041 DEBUG:teuthology.orchestra.run.smithi152:> sudo hostname smithi152 2023-11-11T00:11:57.057 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.15.169 smithi169.front.sepia.ceph.com smithi169 2023-11-11T00:11:57.057 DEBUG:teuthology.orchestra.run.smithi169:> sudo hostname smithi169 2023-11-11T00:11:57.154 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i -e 's/smithi152.front.sepia.ceph.com/smithi152/g' /etc/hosts 2023-11-11T00:11:57.181 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i -e 's/172.21.15.152/172.21.15.152/g' /etc/hosts 2023-11-11T00:11:57.193 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i -e 's/smithi169.front.sepia.ceph.com/smithi169/g' /etc/hosts 2023-11-11T00:11:57.222 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i -e 's/172.21.15.169/172.21.15.169/g' /etc/hosts 2023-11-11T00:11:57.249 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i -e 's/smithi152.front.sepia.ceph.com/smithi152/g' /etc/hostname 2023-11-11T00:11:57.289 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i -e 's/smithi169.front.sepia.ceph.com/smithi169/g' /etc/hostname 2023-11-11T00:11:57.316 DEBUG:teuthology.orchestra.run.smithi152:> sudo hostname smithi152 2023-11-11T00:11:57.355 DEBUG:teuthology.orchestra.run.smithi169:> sudo hostname smithi169 2023-11-11T00:11:57.382 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2023-11-11T00:11:57.422 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2023-11-11T00:11:57.436 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2023-11-11T00:11:57.437 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="9" 2023-11-11T00:11:57.437 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2023-11-11T00:11:57.437 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2023-11-11T00:11:57.437 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="9" 2023-11-11T00:11:57.438 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el9" 2023-11-11T00:11:57.438 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-11T00:11:57.438 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2023-11-11T00:11:57.438 INFO:teuthology.orchestra.run.smithi152.stdout:LOGO="fedora-logo-icon" 2023-11-11T00:11:57.438 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-11T00:11:57.439 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2023-11-11T00:11:57.439 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-11T00:11:57.439 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-11T00:11:57.439 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-11T00:11:57.440 INFO:teuthology.provision.fog.smithi152:Deploy complete! 2023-11-11T00:11:57.441 DEBUG:teuthology.parallel:result is None 2023-11-11T00:11:57.477 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="CentOS Stream" 2023-11-11T00:11:57.477 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="9" 2023-11-11T00:11:57.477 INFO:teuthology.orchestra.run.smithi169.stdout:ID="centos" 2023-11-11T00:11:57.478 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="rhel fedora" 2023-11-11T00:11:57.478 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="9" 2023-11-11T00:11:57.478 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el9" 2023-11-11T00:11:57.478 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-11T00:11:57.478 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31" 2023-11-11T00:11:57.479 INFO:teuthology.orchestra.run.smithi169.stdout:LOGO="fedora-logo-icon" 2023-11-11T00:11:57.479 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-11T00:11:57.479 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://centos.org/" 2023-11-11T00:11:57.479 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-11T00:11:57.479 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-11T00:11:57.480 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-11T00:11:57.480 INFO:teuthology.provision.fog.smithi169:Deploy complete! 2023-11-11T00:11:57.481 DEBUG:teuthology.parallel:result is None 2023-11-11T00:11:57.639 INFO:teuthology.lock.ops:Checking smithi152.front.sepia.ceph.com 2023-11-11T00:11:57.640 INFO:teuthology.lock.ops:New key found. Updating... 2023-11-11T00:11:57.666 INFO:teuthology.lock.ops:Checking smithi169.front.sepia.ceph.com 2023-11-11T00:11:57.666 INFO:teuthology.lock.ops:New key found. Updating... 2023-11-11T00:11:57.692 INFO:teuthology.lock.ops:Updating [smithi152.front.sepia.ceph.com]: set os type and version on server 2023-11-11T00:11:57.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2023-11-11T00:11:57.694 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:11:57.777 DEBUG:teuthology.orchestra.run.smithi152:> true 2023-11-11T00:11:57.844 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2023-11-11T00:11:57.844 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2023-11-11T00:11:57.898 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2023-11-11T00:11:57.899 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2023-11-11T00:11:57.952 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2023-11-11T00:11:57.952 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="9" 2023-11-11T00:11:57.952 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2023-11-11T00:11:57.953 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2023-11-11T00:11:57.953 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="9" 2023-11-11T00:11:57.953 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el9" 2023-11-11T00:11:57.953 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-11T00:11:57.953 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2023-11-11T00:11:57.954 INFO:teuthology.orchestra.run.smithi152.stdout:LOGO="fedora-logo-icon" 2023-11-11T00:11:57.954 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-11T00:11:57.954 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2023-11-11T00:11:57.954 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-11T00:11:57.954 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-11T00:11:57.955 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-11T00:11:57.955 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2023-11-11T00:11:57.974 INFO:teuthology.lock.ops:Updating [smithi169.front.sepia.ceph.com]: set os type and version on server 2023-11-11T00:11:57.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-11-11T00:11:57.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-11T00:11:58.057 DEBUG:teuthology.orchestra.run.smithi169:> true 2023-11-11T00:11:58.126 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-11-11T00:11:58.126 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2023-11-11T00:11:58.180 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2023-11-11T00:11:58.180 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2023-11-11T00:11:58.234 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="CentOS Stream" 2023-11-11T00:11:58.234 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="9" 2023-11-11T00:11:58.234 INFO:teuthology.orchestra.run.smithi169.stdout:ID="centos" 2023-11-11T00:11:58.235 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="rhel fedora" 2023-11-11T00:11:58.235 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="9" 2023-11-11T00:11:58.235 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el9" 2023-11-11T00:11:58.235 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-11T00:11:58.236 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31" 2023-11-11T00:11:58.236 INFO:teuthology.orchestra.run.smithi169.stdout:LOGO="fedora-logo-icon" 2023-11-11T00:11:58.236 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-11T00:11:58.236 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://centos.org/" 2023-11-11T00:11:58.236 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-11T00:11:58.237 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-11T00:11:58.237 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-11T00:11:58.237 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2023-11-11T00:11:58.258 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-11T00:11:58.353 INFO:teuthology.dispatcher.supervisor:Running job 7454972 2023-11-11T00:11:58.353 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-11-10_19:52:04-fs-wip-yuri5-testing-2023-11-10-0828-distro-default-smithi/7454972 --name yuriw-2023-11-10_19:52:04-fs-wip-yuri5-testing-2023-11-10-0828-distro-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/yuriw-2023-11-10_19:52:04-fs-wip-yuri5-testing-2023-11-10-0828-distro-default-smithi/7454972/orig.config.yaml 2023-11-11T00:11:58.356 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-11-10_19:52:04-fs-wip-yuri5-testing-2023-11-10-0828-distro-default-smithi/7454972 2023-11-11T00:11:58.357 INFO:teuthology.dispatcher.supervisor:Job PID: 3719956 2023-11-11T00:11:58.357 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-11-11T00:13:58.455 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-11T00:15:58.627 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-11T00:17:58.795 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-11T00:19:58.971 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-11T00:21:59.140 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-11T00:21:59.203 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120 2023-11-11T00:21:59.333 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2023-11-11T00:21:59.333 INFO:teuthology.nuke:Checking targets against current locks 2023-11-11T00:21:59.382 DEBUG:teuthology.nuke:shortname: smithi152 2023-11-11T00:21:59.383 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-11T00:21:59.406 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-10_19:52:04-fs-wip-yuri5-testing-2023-11-10-0828-distro-default-smithi/7454972', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-11 00:06:42.346779', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHjp4GLz9eiIxhLkf4o6wx+kAxqJSrmx/BOKVWGFoCCyxodNjINoOWDPijjthZuABDYtgWklU7ui1eKkRbT4j/Q='} 2023-11-11T00:21:59.420 DEBUG:teuthology.nuke:shortname: smithi169 2023-11-11T00:21:59.420 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-11T00:21:59.445 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-10_19:52:04-fs-wip-yuri5-testing-2023-11-10-0828-distro-default-smithi/7454972', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-11 00:06:42.344799', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHMTTOPxU/vdxUXjzsL2qqPlSlBaH6MI/3X8+7QjikvQLTOCVRmUY5s8j8szqjar+ADAxMWtFrXsMku4vYUFWx4='} 2023-11-11T00:21:59.459 INFO:teuthology.orchestra.console.smithi152:Power off 2023-11-11T00:21:59.460 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-11T00:21:59.481 DEBUG:teuthology.orchestra.console.smithi152:power off output: Chassis Power Control: Down/Off 2023-11-11T00:21:59.482 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-11T00:21:59.503 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is off 2023-11-11T00:21:59.525 INFO:teuthology.orchestra.console.smithi169:Power off 2023-11-11T00:21:59.526 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-11T00:21:59.549 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2023-11-11T00:21:59.550 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-11T00:21:59.574 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2023-11-11T00:21:59.604 INFO:teuthology.orchestra.console.smithi152:Power off completed 2023-11-11T00:21:59.676 INFO:teuthology.orchestra.console.smithi169:Power off completed 2023-11-11T00:21:59.760 INFO:teuthology.lock.ops:unlocked: smithi152.front.sepia.ceph.com 2023-11-11T00:21:59.913 INFO:teuthology.lock.ops:unlocked: smithi169.front.sepia.ceph.com