2023-08-30T09:24:13.287 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-30T09:24:13.289 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-30T09:24:13.426 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb65f9211f0>, signals=[15]) 2023-08-30T09:24:13.427 INFO:teuthology.lock.ops:Start node 'smithi083.front.sepia.ceph.com' reimaging 2023-08-30T09:24:13.428 INFO:teuthology.lock.ops:Updating [smithi083.front.sepia.ceph.com]: reset os type and version on server 2023-08-30T09:24:13.428 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2023-08-30T09:24:13.448 INFO:teuthology.lock.ops:Start node 'smithi121.front.sepia.ceph.com' reimaging 2023-08-30T09:24:13.449 INFO:teuthology.lock.ops:Updating [smithi121.front.sepia.ceph.com]: reset os type and version on server 2023-08-30T09:24:13.449 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2023-08-30T09:24:13.451 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /host {"name": "smithi083"} 2023-08-30T09:24:13.472 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /host {"name": "smithi121"} 2023-08-30T09:24:13.520 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /image {"name": "smithi_centos_9.stream"} 2023-08-30T09:24:13.541 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /image {"name": "smithi_centos_9.stream"} 2023-08-30T09:24:13.566 DEBUG:teuthology.provision.fog.smithi083:Requesting image smithi_centos_9.stream (ID 10) 2023-08-30T09:24:13.566 DEBUG:teuthology.provision.fog.smithi083:FOG request: PUT /host/188 {"imageID": 10} 2023-08-30T09:24:13.583 DEBUG:teuthology.provision.fog.smithi121:Requesting image smithi_centos_9.stream (ID 10) 2023-08-30T09:24:13.583 DEBUG:teuthology.provision.fog.smithi121:FOG request: PUT /host/226 {"imageID": 10} 2023-08-30T09:24:13.617 INFO:teuthology.provision.fog.smithi083:Scheduling deploy of centos 9.stream 2023-08-30T09:24:13.617 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None 2023-08-30T09:24:13.632 INFO:teuthology.provision.fog.smithi121:Scheduling deploy of centos 9.stream 2023-08-30T09:24:13.633 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-30T09:24:13.705 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /tasktype {"name": "deploy"} 2023-08-30T09:24:13.713 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /tasktype {"name": "deploy"} 2023-08-30T09:24:13.744 DEBUG:teuthology.provision.fog.smithi121:FOG request: POST /host/226/task {"taskTypeID": 1} 2023-08-30T09:24:13.747 DEBUG:teuthology.provision.fog.smithi083:FOG request: POST /host/188/task {"taskTypeID": 1} 2023-08-30T09:24:13.806 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None 2023-08-30T09:24:13.820 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-30T09:24:13.921 INFO:teuthology.orchestra.console.smithi083:Power off 2023-08-30T09:24:13.922 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-30T09:24:13.936 INFO:teuthology.orchestra.console.smithi121:Power off 2023-08-30T09:24:13.937 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-30T09:24:13.947 DEBUG:teuthology.orchestra.console.smithi083:power off output: Chassis Power Control: Down/Off 2023-08-30T09:24:13.948 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-30T09:24:13.959 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2023-08-30T09:24:13.960 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-30T09:24:13.972 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2023-08-30T09:24:13.981 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2023-08-30T09:24:17.978 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-30T09:24:17.986 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-30T09:24:18.089 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2023-08-30T09:24:18.096 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2023-08-30T09:24:22.094 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-30T09:24:22.101 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-30T09:24:22.203 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2023-08-30T09:24:22.210 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2023-08-30T09:24:26.206 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-30T09:24:26.213 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-30T09:24:26.315 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is off 2023-08-30T09:24:26.321 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2023-08-30T09:24:26.417 INFO:teuthology.orchestra.console.smithi083:Power off completed 2023-08-30T09:24:26.422 INFO:teuthology.orchestra.console.smithi121:Power off completed 2023-08-30T09:24:26.518 INFO:teuthology.orchestra.console.smithi083:Power on 2023-08-30T09:24:26.518 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-08-30T09:24:26.525 INFO:teuthology.orchestra.console.smithi121:Power on 2023-08-30T09:24:26.525 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-08-30T09:24:26.541 DEBUG:teuthology.orchestra.console.smithi083:power on output: Chassis Power Control: Up/On 2023-08-30T09:24:26.541 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-30T09:24:26.551 DEBUG:teuthology.orchestra.console.smithi121:power on output: Chassis Power Control: Up/On 2023-08-30T09:24:26.551 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-30T09:24:26.562 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is off 2023-08-30T09:24:26.572 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2023-08-30T09:24:30.568 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-30T09:24:30.575 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-30T09:24:30.677 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2023-08-30T09:24:30.684 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2023-08-30T09:24:30.779 INFO:teuthology.orchestra.console.smithi083:Power on completed 2023-08-30T09:24:30.880 INFO:teuthology.provision.fog.smithi083:Waiting for deploy to finish 2023-08-30T09:24:30.881 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None 2023-08-30T09:24:34.686 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-30T09:24:34.907 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2023-08-30T09:24:35.008 INFO:teuthology.orchestra.console.smithi121:Power on completed 2023-08-30T09:24:35.109 INFO:teuthology.provision.fog.smithi121:Waiting for deploy to finish 2023-08-30T09:24:35.110 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-30T09:24:45.966 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None 2023-08-30T09:24:50.190 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-30T09:25:01.046 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None 2023-08-30T09:25:05.255 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-30T09:25:16.118 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None 2023-08-30T09:25:20.326 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-30T09:25:31.194 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None 2023-08-30T09:25:35.406 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-30T09:25:46.290 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None 2023-08-30T09:25:50.494 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-30T09:26:01.370 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None 2023-08-30T09:26:05.566 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-30T09:26:16.470 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None 2023-08-30T09:26:20.666 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-30T09:26:31.575 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None 2023-08-30T09:26:35.750 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-30T09:26:46.674 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None 2023-08-30T09:26:50.822 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-30T09:27:01.777 DEBUG:teuthology.provision.fog.smithi083:FOG request: GET /task/active None 2023-08-30T09:27:01.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:27:04.926 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83 2023-08-30T09:27:05.901 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-30T09:27:05.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:27:09.050 WARNING:teuthology.provision.fog.smithi121:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-08-30T09:27:10.930 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:27:14.010 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83 2023-08-30T09:27:15.052 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:27:18.110 WARNING:teuthology.provision.fog.smithi121:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-08-30T09:27:20.013 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:27:23.070 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83 2023-08-30T09:27:24.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:27:27.162 WARNING:teuthology.provision.fog.smithi121:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-08-30T09:27:29.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:27:32.154 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83 2023-08-30T09:27:33.165 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:27:36.222 WARNING:teuthology.provision.fog.smithi121:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-08-30T09:27:38.156 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:27:41.210 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83 2023-08-30T09:27:42.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:27:45.306 WARNING:teuthology.provision.fog.smithi121:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-08-30T09:27:47.214 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:27:50.270 WARNING:teuthology.provision.fog.smithi083:[Errno None] Unable to connect to port 22 on 172.21.15.83 2023-08-30T09:27:51.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:27:54.362 WARNING:teuthology.provision.fog.smithi121:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-08-30T09:27:56.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:28:00.366 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:28:03.422 WARNING:teuthology.provision.fog.smithi121:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-08-30T09:28:09.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:28:27.802 WARNING:teuthology.provision.fog.smithi121:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-08-30T09:28:33.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:28:36.858 WARNING:teuthology.provision.fog.smithi121:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-08-30T09:28:42.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:28:45.914 WARNING:teuthology.provision.fog.smithi121:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-08-30T09:28:51.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:28:52.114 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-08-30T09:28:52.115 DEBUG:teuthology.orchestra.run.smithi121:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-08-30T09:28:56.276 WARNING:teuthology.provision.fog.smithi083:timed out 2023-08-30T09:29:02.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:29:02.467 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-08-30T09:29:02.468 DEBUG:teuthology.orchestra.run.smithi083:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-08-30T09:29:02.856 INFO:teuthology.provision.fog.smithi083:Node is ready 2023-08-30T09:29:02.857 DEBUG:teuthology.orchestra.run.smithi083:> hostname 2023-08-30T09:29:02.871 INFO:teuthology.orchestra.run.smithi083.stdout:smithi083.front.sepia.ceph.com 2023-08-30T09:29:02.872 DEBUG:teuthology.orchestra.run.smithi083:> grep smithi083.front.sepia.ceph.com /etc/hosts 2023-08-30T09:29:02.926 INFO:teuthology.orchestra.run.smithi083.stdout:172.21.15.83 smithi083.front.sepia.ceph.com smithi083 2023-08-30T09:29:02.927 DEBUG:teuthology.orchestra.run.smithi083:> sudo hostname smithi083 2023-08-30T09:29:03.024 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i -e 's/smithi083.front.sepia.ceph.com/smithi083/g' /etc/hosts 2023-08-30T09:29:03.052 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i -e 's/172.21.15.83/172.21.15.83/g' /etc/hosts 2023-08-30T09:29:03.120 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i -e 's/smithi083.front.sepia.ceph.com/smithi083/g' /etc/hostname 2023-08-30T09:29:03.187 DEBUG:teuthology.orchestra.run.smithi083:> sudo hostname smithi083 2023-08-30T09:29:03.254 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2023-08-30T09:29:03.307 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Stream" 2023-08-30T09:29:03.308 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="9" 2023-08-30T09:29:03.308 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos" 2023-08-30T09:29:03.308 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora" 2023-08-30T09:29:03.308 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="9" 2023-08-30T09:29:03.308 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el9" 2023-08-30T09:29:03.309 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Stream 9" 2023-08-30T09:29:03.309 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31" 2023-08-30T09:29:03.309 INFO:teuthology.orchestra.run.smithi083.stdout:LOGO="fedora-logo-icon" 2023-08-30T09:29:03.309 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-08-30T09:29:03.309 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/" 2023-08-30T09:29:03.310 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-30T09:29:03.310 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-08-30T09:29:03.310 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-08-30T09:29:03.311 INFO:teuthology.provision.fog.smithi083:Deploy complete! 2023-08-30T09:29:03.346 DEBUG:teuthology.parallel:result is None 2023-08-30T09:29:12.737 INFO:teuthology.provision.fog.smithi121:Node is ready 2023-08-30T09:29:12.738 DEBUG:teuthology.orchestra.run.smithi121:> hostname 2023-08-30T09:29:12.757 INFO:teuthology.orchestra.run.smithi121.stdout:smithi121.front.sepia.ceph.com 2023-08-30T09:29:12.757 DEBUG:teuthology.orchestra.run.smithi121:> grep smithi121.front.sepia.ceph.com /etc/hosts 2023-08-30T09:29:12.815 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121 smithi121.front.sepia.ceph.com smithi121 2023-08-30T09:29:12.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo hostname smithi121 2023-08-30T09:29:12.950 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i -e 's/smithi121.front.sepia.ceph.com/smithi121/g' /etc/hosts 2023-08-30T09:29:12.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i -e 's/172.21.15.121/172.21.15.121/g' /etc/hosts 2023-08-30T09:29:13.052 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i -e 's/smithi121.front.sepia.ceph.com/smithi121/g' /etc/hostname 2023-08-30T09:29:13.120 DEBUG:teuthology.orchestra.run.smithi121:> sudo hostname smithi121 2023-08-30T09:29:13.187 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2023-08-30T09:29:13.242 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2023-08-30T09:29:13.242 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="9" 2023-08-30T09:29:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2023-08-30T09:29:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2023-08-30T09:29:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="9" 2023-08-30T09:29:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el9" 2023-08-30T09:29:13.243 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 9" 2023-08-30T09:29:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2023-08-30T09:29:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:LOGO="fedora-logo-icon" 2023-08-30T09:29:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-08-30T09:29:13.244 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2023-08-30T09:29:13.245 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-30T09:29:13.245 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-08-30T09:29:13.245 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-08-30T09:29:13.246 INFO:teuthology.provision.fog.smithi121:Deploy complete! 2023-08-30T09:29:13.246 DEBUG:teuthology.parallel:result is None 2023-08-30T09:29:13.402 INFO:teuthology.lock.ops:Checking smithi083.front.sepia.ceph.com 2023-08-30T09:29:13.403 INFO:teuthology.lock.ops:New key found. Updating... 2023-08-30T09:29:13.426 INFO:teuthology.lock.ops:Checking smithi121.front.sepia.ceph.com 2023-08-30T09:29:13.426 INFO:teuthology.lock.ops:New key found. Updating... 2023-08-30T09:29:13.444 INFO:teuthology.lock.ops:Updating [smithi083.front.sepia.ceph.com]: set os type and version on server 2023-08-30T09:29:13.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2023-08-30T09:29:13.445 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:29:13.529 DEBUG:teuthology.orchestra.run.smithi083:> true 2023-08-30T09:29:13.691 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2023-08-30T09:29:13.692 DEBUG:teuthology.orchestra.run.smithi083:> uname -m 2023-08-30T09:29:13.745 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64 2023-08-30T09:29:13.746 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2023-08-30T09:29:13.799 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Stream" 2023-08-30T09:29:13.799 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="9" 2023-08-30T09:29:13.800 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos" 2023-08-30T09:29:13.800 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora" 2023-08-30T09:29:13.800 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="9" 2023-08-30T09:29:13.800 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el9" 2023-08-30T09:29:13.800 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Stream 9" 2023-08-30T09:29:13.801 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31" 2023-08-30T09:29:13.801 INFO:teuthology.orchestra.run.smithi083.stdout:LOGO="fedora-logo-icon" 2023-08-30T09:29:13.801 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-08-30T09:29:13.801 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/" 2023-08-30T09:29:13.801 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-30T09:29:13.802 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-08-30T09:29:13.802 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-08-30T09:29:13.802 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2023-08-30T09:29:13.826 INFO:teuthology.lock.ops:Updating [smithi121.front.sepia.ceph.com]: set os type and version on server 2023-08-30T09:29:13.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-08-30T09:29:13.827 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-30T09:29:13.910 DEBUG:teuthology.orchestra.run.smithi121:> true 2023-08-30T09:29:13.936 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-08-30T09:29:13.937 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2023-08-30T09:29:13.992 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2023-08-30T09:29:13.992 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2023-08-30T09:29:14.046 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2023-08-30T09:29:14.046 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="9" 2023-08-30T09:29:14.047 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2023-08-30T09:29:14.047 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2023-08-30T09:29:14.047 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="9" 2023-08-30T09:29:14.047 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el9" 2023-08-30T09:29:14.047 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 9" 2023-08-30T09:29:14.048 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2023-08-30T09:29:14.048 INFO:teuthology.orchestra.run.smithi121.stdout:LOGO="fedora-logo-icon" 2023-08-30T09:29:14.048 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-08-30T09:29:14.048 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2023-08-30T09:29:14.048 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-30T09:29:14.049 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-08-30T09:29:14.049 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-08-30T09:29:14.049 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2023-08-30T09:29:14.076 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-30T09:29:14.189 INFO:teuthology.dispatcher.supervisor:Running job 7382636 2023-08-30T09:29:14.190 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/virtualenv/bin/teuthology -v --owner scheduled_mkogan@teuthology --archive /home/teuthworker/archive/mkogan-2023-08-29_15:26:33-rgw-rgw-wip-t62141-valg-supp_i004-distro-default-smithi/7382636 --name mkogan-2023-08-29_15:26:33-rgw-rgw-wip-t62141-valg-supp_i004-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_latest} tasks/test_multi} -- /home/teuthworker/archive/mkogan-2023-08-29_15:26:33-rgw-rgw-wip-t62141-valg-supp_i004-distro-default-smithi/7382636/orig.config.yaml 2023-08-30T09:29:14.193 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/mkogan-2023-08-29_15:26:33-rgw-rgw-wip-t62141-valg-supp_i004-distro-default-smithi/7382636 2023-08-30T09:29:14.194 INFO:teuthology.dispatcher.supervisor:Job PID: 36470 2023-08-30T09:29:14.194 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-08-30T09:31:14.288 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-30T09:31:14.350 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120 2023-08-30T09:31:14.452 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2023-08-30T09:31:14.452 INFO:teuthology.nuke:Checking targets against current locks 2023-08-30T09:31:14.495 DEBUG:teuthology.nuke:shortname: smithi083 2023-08-30T09:31:14.495 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-30T09:31:14.516 DEBUG:teuthology.nuke:shortname: smithi121 2023-08-30T09:31:14.516 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-30T09:31:14.519 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-08-29_15:26:33-rgw-rgw-wip-t62141-valg-supp_i004-distro-default-smithi/7382636', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-30 09:24:12.491630', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBnP+V4SEDxxsebpDBP0czbqK93BcGTKlKq8y1WaxN6gHTAw46d/jx20x9UugMdKxF9gWEv7uIINL/O+bEy2I+s='} 2023-08-30T09:31:14.539 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-08-29_15:26:33-rgw-rgw-wip-t62141-valg-supp_i004-distro-default-smithi/7382636', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-30 09:24:12.489620', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF+Vmtvd5b4d1NCJX0TNfcVWVDJLMVY+WdUBNXddQO02TeZFegIyJk607WAPYtdmDkKg8f76KEvpssiO6vf4Go0='} 2023-08-30T09:31:14.558 INFO:teuthology.orchestra.console.smithi083:Power off 2023-08-30T09:31:14.559 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-30T09:31:14.579 DEBUG:teuthology.orchestra.console.smithi083:power off output: Chassis Power Control: Down/Off 2023-08-30T09:31:14.579 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-30T09:31:14.595 INFO:teuthology.orchestra.console.smithi121:Power off 2023-08-30T09:31:14.596 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-30T09:31:14.604 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is off 2023-08-30T09:31:14.615 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2023-08-30T09:31:14.615 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-30T09:31:14.636 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2023-08-30T09:31:14.705 INFO:teuthology.orchestra.console.smithi083:Power off completed 2023-08-30T09:31:14.738 INFO:teuthology.orchestra.console.smithi121:Power off completed 2023-08-30T09:31:14.859 INFO:teuthology.lock.ops:unlocked: smithi083.front.sepia.ceph.com 2023-08-30T09:31:14.883 INFO:teuthology.lock.ops:unlocked: smithi121.front.sepia.ceph.com