2023-11-08T01:27:36.491 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-08T01:27:36.492 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T01:27:36.640 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f69bf260670>, signals=[15]) 2023-11-08T01:27:36.641 INFO:teuthology.lock.ops:Start node 'smithi149.front.sepia.ceph.com' reimaging 2023-11-08T01:27:36.642 INFO:teuthology.lock.ops:Updating [smithi149.front.sepia.ceph.com]: reset os type and version on server 2023-11-08T01:27:36.642 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2023-11-08T01:27:36.672 INFO:teuthology.lock.ops:Start node 'smithi164.front.sepia.ceph.com' reimaging 2023-11-08T01:27:36.673 INFO:teuthology.lock.ops:Updating [smithi164.front.sepia.ceph.com]: reset os type and version on server 2023-11-08T01:27:36.673 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2023-11-08T01:27:36.674 DEBUG:teuthology.provision.fog.smithi149:FOG request: GET /host {"name": "smithi149"} 2023-11-08T01:27:36.697 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /host {"name": "smithi164"} 2023-11-08T01:27:36.745 DEBUG:teuthology.provision.fog.smithi149:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-11-08T01:27:36.772 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-11-08T01:27:36.785 DEBUG:teuthology.provision.fog.smithi149:Requesting image smithi_centos_8.stream (ID 3) 2023-11-08T01:27:36.785 DEBUG:teuthology.provision.fog.smithi149:FOG request: PUT /host/254 {"imageID": 3} 2023-11-08T01:27:36.818 DEBUG:teuthology.provision.fog.smithi164:Requesting image smithi_centos_8.stream (ID 3) 2023-11-08T01:27:36.818 DEBUG:teuthology.provision.fog.smithi164:FOG request: PUT /host/269 {"imageID": 3} 2023-11-08T01:27:36.831 INFO:teuthology.provision.fog.smithi149:Scheduling deploy of centos 8.stream 2023-11-08T01:27:36.831 DEBUG:teuthology.provision.fog.smithi149:FOG request: GET /task/active None 2023-11-08T01:27:36.866 INFO:teuthology.provision.fog.smithi164:Scheduling deploy of centos 8.stream 2023-11-08T01:27:36.867 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None 2023-11-08T01:27:36.883 DEBUG:teuthology.provision.fog.smithi149:FOG request: GET /tasktype {"name": "deploy"} 2023-11-08T01:27:36.917 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /tasktype {"name": "deploy"} 2023-11-08T01:27:36.920 DEBUG:teuthology.provision.fog.smithi149:FOG request: POST /host/254/task {"taskTypeID": 1} 2023-11-08T01:27:36.954 DEBUG:teuthology.provision.fog.smithi164:FOG request: POST /host/269/task {"taskTypeID": 1} 2023-11-08T01:27:36.991 DEBUG:teuthology.provision.fog.smithi149:FOG request: GET /task/active None 2023-11-08T01:27:37.001 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None 2023-11-08T01:27:37.075 INFO:teuthology.orchestra.console.smithi149:Power off 2023-11-08T01:27:37.076 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-08T01:27:37.088 INFO:teuthology.orchestra.console.smithi164:Power off 2023-11-08T01:27:37.089 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-08T01:27:37.097 DEBUG:teuthology.orchestra.console.smithi149:power off output: Chassis Power Control: Down/Off 2023-11-08T01:27:37.097 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T01:27:37.108 DEBUG:teuthology.orchestra.console.smithi164:power off output: Chassis Power Control: Down/Off 2023-11-08T01:27:37.109 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T01:27:37.119 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is off 2023-11-08T01:27:37.150 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is off 2023-11-08T01:27:37.221 INFO:teuthology.orchestra.console.smithi149:Power off completed 2023-11-08T01:27:37.252 INFO:teuthology.orchestra.console.smithi164:Power off completed 2023-11-08T01:27:37.322 INFO:teuthology.orchestra.console.smithi149:Power on 2023-11-08T01:27:37.323 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-11-08T01:27:37.347 DEBUG:teuthology.orchestra.console.smithi149:power on output: Chassis Power Control: Up/On 2023-11-08T01:27:37.348 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T01:27:37.356 INFO:teuthology.orchestra.console.smithi164:Power on 2023-11-08T01:27:37.356 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-11-08T01:27:37.375 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is off 2023-11-08T01:27:37.376 DEBUG:teuthology.orchestra.console.smithi164:power on output: Chassis Power Control: Up/On 2023-11-08T01:27:37.376 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T01:27:37.397 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is off 2023-11-08T01:27:41.378 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T01:27:41.399 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T01:27:41.488 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is off 2023-11-08T01:27:41.509 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2023-11-08T01:27:41.610 INFO:teuthology.orchestra.console.smithi164:Power on completed 2023-11-08T01:27:41.711 INFO:teuthology.provision.fog.smithi164:Waiting for deploy to finish 2023-11-08T01:27:41.711 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None 2023-11-08T01:27:45.490 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T01:27:45.600 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2023-11-08T01:27:45.701 INFO:teuthology.orchestra.console.smithi149:Power on completed 2023-11-08T01:27:45.801 INFO:teuthology.provision.fog.smithi149:Waiting for deploy to finish 2023-11-08T01:27:45.802 DEBUG:teuthology.provision.fog.smithi149:FOG request: GET /task/active None 2023-11-08T01:27:56.786 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None 2023-11-08T01:28:00.873 DEBUG:teuthology.provision.fog.smithi149:FOG request: GET /task/active None 2023-11-08T01:28:11.856 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None 2023-11-08T01:28:15.938 DEBUG:teuthology.provision.fog.smithi149:FOG request: GET /task/active None 2023-11-08T01:28:26.930 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None 2023-11-08T01:28:31.006 DEBUG:teuthology.provision.fog.smithi149:FOG request: GET /task/active None 2023-11-08T01:28:42.014 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None 2023-11-08T01:28:46.079 DEBUG:teuthology.provision.fog.smithi149:FOG request: GET /task/active None 2023-11-08T01:28:57.093 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None 2023-11-08T01:29:01.142 DEBUG:teuthology.provision.fog.smithi149:FOG request: GET /task/active None 2023-11-08T01:29:12.166 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None 2023-11-08T01:29:16.206 DEBUG:teuthology.provision.fog.smithi149:FOG request: GET /task/active None 2023-11-08T01:29:27.246 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None 2023-11-08T01:29:31.274 DEBUG:teuthology.provision.fog.smithi149:FOG request: GET /task/active None 2023-11-08T01:29:42.310 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None 2023-11-08T01:29:46.334 DEBUG:teuthology.provision.fog.smithi149:FOG request: GET /task/active None 2023-11-08T01:29:57.382 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None 2023-11-08T01:30:01.398 DEBUG:teuthology.provision.fog.smithi149:FOG request: GET /task/active None 2023-11-08T01:30:12.454 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None 2023-11-08T01:30:16.454 DEBUG:teuthology.provision.fog.smithi149:FOG request: GET /task/active None 2023-11-08T01:30:27.522 DEBUG:teuthology.provision.fog.smithi164:FOG request: GET /task/active None 2023-11-08T01:30:27.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T01:30:30.650 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164 2023-11-08T01:30:31.515 DEBUG:teuthology.provision.fog.smithi149:FOG request: GET /task/active None 2023-11-08T01:30:31.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T01:30:34.650 WARNING:teuthology.provision.fog.smithi149:[Errno None] Unable to connect to port 22 on 172.21.15.149 2023-11-08T01:30:36.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T01:30:39.710 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164 2023-11-08T01:30:40.652 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T01:30:43.706 WARNING:teuthology.provision.fog.smithi149:[Errno None] Unable to connect to port 22 on 172.21.15.149 2023-11-08T01:30:45.714 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T01:30:48.794 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164 2023-11-08T01:30:49.709 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T01:30:52.766 WARNING:teuthology.provision.fog.smithi149:[Errno None] Unable to connect to port 22 on 172.21.15.149 2023-11-08T01:30:54.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T01:30:57.854 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164 2023-11-08T01:30:58.768 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T01:31:01.854 WARNING:teuthology.provision.fog.smithi149:[Errno None] Unable to connect to port 22 on 172.21.15.149 2023-11-08T01:31:03.857 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T01:31:06.938 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164 2023-11-08T01:31:07.856 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T01:31:10.938 WARNING:teuthology.provision.fog.smithi149:[Errno None] Unable to connect to port 22 on 172.21.15.149 2023-11-08T01:31:12.942 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T01:31:15.994 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164 2023-11-08T01:31:16.940 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T01:31:19.994 WARNING:teuthology.provision.fog.smithi149:[Errno None] Unable to connect to port 22 on 172.21.15.149 2023-11-08T01:31:21.998 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T01:31:25.050 WARNING:teuthology.provision.fog.smithi164:[Errno None] Unable to connect to port 22 on 172.21.15.164 2023-11-08T01:31:25.997 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T01:31:31.056 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T01:32:02.772 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-11-08T01:32:02.773 DEBUG:teuthology.orchestra.run.smithi164:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-11-08T01:32:26.000 WARNING:teuthology.provision.fog.smithi149:timed out 2023-11-08T01:32:28.236 INFO:teuthology.provision.fog.smithi164:Node is ready 2023-11-08T01:32:28.237 DEBUG:teuthology.orchestra.run.smithi164:> hostname 2023-11-08T01:32:28.261 INFO:teuthology.orchestra.run.smithi164.stdout:smithi164.front.sepia.ceph.com 2023-11-08T01:32:28.262 DEBUG:teuthology.orchestra.run.smithi164:> grep smithi164.front.sepia.ceph.com /etc/hosts 2023-11-08T01:32:28.322 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.15.164 smithi164.front.sepia.ceph.com smithi164 2023-11-08T01:32:28.322 DEBUG:teuthology.orchestra.run.smithi164:> sudo hostname smithi164 2023-11-08T01:32:28.452 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i -e 's/smithi164.front.sepia.ceph.com/smithi164/g' /etc/hosts 2023-11-08T01:32:28.484 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i -e 's/172.21.15.164/172.21.15.164/g' /etc/hosts 2023-11-08T01:32:28.555 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i -e 's/smithi164.front.sepia.ceph.com/smithi164/g' /etc/hostname 2023-11-08T01:32:28.623 DEBUG:teuthology.orchestra.run.smithi164:> sudo hostname smithi164 2023-11-08T01:32:28.691 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2023-11-08T01:32:28.749 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="CentOS Stream" 2023-11-08T01:32:28.750 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="8" 2023-11-08T01:32:28.750 INFO:teuthology.orchestra.run.smithi164.stdout:ID="centos" 2023-11-08T01:32:28.750 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="rhel fedora" 2023-11-08T01:32:28.750 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="8" 2023-11-08T01:32:28.751 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el8" 2023-11-08T01:32:28.751 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-08T01:32:28.751 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31" 2023-11-08T01:32:28.751 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-08T01:32:28.751 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://centos.org/" 2023-11-08T01:32:28.752 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-08T01:32:28.752 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-08T01:32:28.752 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-08T01:32:28.753 INFO:teuthology.provision.fog.smithi164:Deploy complete! 2023-11-08T01:32:28.754 DEBUG:teuthology.parallel:result is None 2023-11-08T01:32:32.005 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T01:32:32.270 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-11-08T01:32:32.271 DEBUG:teuthology.orchestra.run.smithi149:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-11-08T01:32:32.549 INFO:teuthology.provision.fog.smithi149:Node is ready 2023-11-08T01:32:32.549 DEBUG:teuthology.orchestra.run.smithi149:> hostname 2023-11-08T01:32:32.575 INFO:teuthology.orchestra.run.smithi149.stdout:smithi149.front.sepia.ceph.com 2023-11-08T01:32:32.575 DEBUG:teuthology.orchestra.run.smithi149:> grep smithi149.front.sepia.ceph.com /etc/hosts 2023-11-08T01:32:32.637 INFO:teuthology.orchestra.run.smithi149.stdout:172.21.15.149 smithi149.front.sepia.ceph.com smithi149 2023-11-08T01:32:32.638 DEBUG:teuthology.orchestra.run.smithi149:> sudo hostname smithi149 2023-11-08T01:32:32.760 DEBUG:teuthology.orchestra.run.smithi149:> sudo sed -i -e 's/smithi149.front.sepia.ceph.com/smithi149/g' /etc/hosts 2023-11-08T01:32:32.791 DEBUG:teuthology.orchestra.run.smithi149:> sudo sed -i -e 's/172.21.15.149/172.21.15.149/g' /etc/hosts 2023-11-08T01:32:32.861 DEBUG:teuthology.orchestra.run.smithi149:> sudo sed -i -e 's/smithi149.front.sepia.ceph.com/smithi149/g' /etc/hostname 2023-11-08T01:32:32.928 DEBUG:teuthology.orchestra.run.smithi149:> sudo hostname smithi149 2023-11-08T01:32:32.995 DEBUG:teuthology.orchestra.run.smithi149:> cat /etc/os-release 2023-11-08T01:32:33.053 INFO:teuthology.orchestra.run.smithi149.stdout:NAME="CentOS Stream" 2023-11-08T01:32:33.054 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION="8" 2023-11-08T01:32:33.054 INFO:teuthology.orchestra.run.smithi149.stdout:ID="centos" 2023-11-08T01:32:33.054 INFO:teuthology.orchestra.run.smithi149.stdout:ID_LIKE="rhel fedora" 2023-11-08T01:32:33.054 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_ID="8" 2023-11-08T01:32:33.055 INFO:teuthology.orchestra.run.smithi149.stdout:PLATFORM_ID="platform:el8" 2023-11-08T01:32:33.055 INFO:teuthology.orchestra.run.smithi149.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-08T01:32:33.055 INFO:teuthology.orchestra.run.smithi149.stdout:ANSI_COLOR="0;31" 2023-11-08T01:32:33.055 INFO:teuthology.orchestra.run.smithi149.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-08T01:32:33.055 INFO:teuthology.orchestra.run.smithi149.stdout:HOME_URL="https://centos.org/" 2023-11-08T01:32:33.056 INFO:teuthology.orchestra.run.smithi149.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-08T01:32:33.056 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-08T01:32:33.056 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-08T01:32:33.056 INFO:teuthology.provision.fog.smithi149:Deploy complete! 2023-11-08T01:32:33.057 DEBUG:teuthology.parallel:result is None 2023-11-08T01:32:33.213 INFO:teuthology.lock.ops:Checking smithi149.front.sepia.ceph.com 2023-11-08T01:32:33.214 INFO:teuthology.lock.ops:New key found. Updating... 2023-11-08T01:32:33.233 INFO:teuthology.lock.ops:Checking smithi164.front.sepia.ceph.com 2023-11-08T01:32:33.233 INFO:teuthology.lock.ops:New key found. Updating... 2023-11-08T01:32:33.250 INFO:teuthology.lock.ops:Updating [smithi149.front.sepia.ceph.com]: set os type and version on server 2023-11-08T01:32:33.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2023-11-08T01:32:33.252 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T01:32:33.320 DEBUG:teuthology.orchestra.run.smithi149:> true 2023-11-08T01:32:33.393 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi149.front.sepia.ceph.com' 2023-11-08T01:32:33.394 DEBUG:teuthology.orchestra.run.smithi149:> uname -m 2023-11-08T01:32:33.451 INFO:teuthology.orchestra.run.smithi149.stdout:x86_64 2023-11-08T01:32:33.452 DEBUG:teuthology.orchestra.run.smithi149:> cat /etc/os-release 2023-11-08T01:32:33.508 INFO:teuthology.orchestra.run.smithi149.stdout:NAME="CentOS Stream" 2023-11-08T01:32:33.508 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION="8" 2023-11-08T01:32:33.509 INFO:teuthology.orchestra.run.smithi149.stdout:ID="centos" 2023-11-08T01:32:33.509 INFO:teuthology.orchestra.run.smithi149.stdout:ID_LIKE="rhel fedora" 2023-11-08T01:32:33.509 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_ID="8" 2023-11-08T01:32:33.509 INFO:teuthology.orchestra.run.smithi149.stdout:PLATFORM_ID="platform:el8" 2023-11-08T01:32:33.510 INFO:teuthology.orchestra.run.smithi149.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-08T01:32:33.510 INFO:teuthology.orchestra.run.smithi149.stdout:ANSI_COLOR="0;31" 2023-11-08T01:32:33.510 INFO:teuthology.orchestra.run.smithi149.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-08T01:32:33.510 INFO:teuthology.orchestra.run.smithi149.stdout:HOME_URL="https://centos.org/" 2023-11-08T01:32:33.510 INFO:teuthology.orchestra.run.smithi149.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-08T01:32:33.511 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-08T01:32:33.511 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-08T01:32:33.512 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2023-11-08T01:32:33.533 INFO:teuthology.lock.ops:Updating [smithi164.front.sepia.ceph.com]: set os type and version on server 2023-11-08T01:32:33.533 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-11-08T01:32:33.534 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T01:32:33.601 DEBUG:teuthology.orchestra.run.smithi164:> true 2023-11-08T01:32:33.680 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2023-11-08T01:32:33.680 DEBUG:teuthology.orchestra.run.smithi164:> uname -m 2023-11-08T01:32:33.741 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2023-11-08T01:32:33.742 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2023-11-08T01:32:33.802 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="CentOS Stream" 2023-11-08T01:32:33.802 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="8" 2023-11-08T01:32:33.802 INFO:teuthology.orchestra.run.smithi164.stdout:ID="centos" 2023-11-08T01:32:33.802 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="rhel fedora" 2023-11-08T01:32:33.803 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="8" 2023-11-08T01:32:33.803 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el8" 2023-11-08T01:32:33.803 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-08T01:32:33.803 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31" 2023-11-08T01:32:33.804 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-08T01:32:33.804 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://centos.org/" 2023-11-08T01:32:33.804 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-08T01:32:33.804 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-08T01:32:33.804 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-08T01:32:33.805 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2023-11-08T01:32:33.825 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T01:32:33.929 INFO:teuthology.dispatcher.supervisor:Running job 7451192 2023-11-08T01:32:33.930 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-11-07_19:34:21-rados-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451192 --name yuriw-2023-11-07_19:34:21-rados-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2023-11-07_19:34:21-rados-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451192/orig.config.yaml 2023-11-08T01:32:33.933 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-11-07_19:34:21-rados-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451192 2023-11-08T01:32:33.934 INFO:teuthology.dispatcher.supervisor:Job PID: 498001 2023-11-08T01:32:33.934 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-11-08T01:34:34.034 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T01:36:34.190 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T01:38:34.308 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T01:40:34.471 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T01:42:34.635 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T01:44:34.807 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T01:46:34.964 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T01:48:35.127 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T01:50:35.287 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T01:52:35.463 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T01:54:35.639 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T01:56:35.811 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T01:58:35.973 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:00:36.139 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:02:36.306 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:04:36.480 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:06:36.656 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:08:36.827 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:10:36.981 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:12:37.155 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:14:37.336 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:16:37.505 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:18:37.687 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:20:37.859 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:22:37.999 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:24:38.174 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:26:38.347 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:28:38.522 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:30:38.691 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:32:38.851 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:34:39.015 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:36:39.191 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:38:39.360 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:40:39.531 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:42:39.695 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:44:39.863 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:46:40.011 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:48:40.180 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T02:48:40.235 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120 2023-11-08T02:48:40.435 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2023-11-08T02:48:40.436 INFO:teuthology.nuke:Checking targets against current locks 2023-11-08T02:48:40.485 DEBUG:teuthology.nuke:shortname: smithi149 2023-11-08T02:48:40.485 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-08T02:48:40.506 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-07_19:34:21-rados-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451192', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-08 01:27:35.652834', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPsc6Jt4JCpE+uqBWeTNxncyWOtQ6FqFVQ10nU6cjGwqgmKagfFt+vEszKOW0b95lWh6953VtFsAzpdkA4WGpYM='} 2023-11-08T02:48:40.510 DEBUG:teuthology.nuke:shortname: smithi164 2023-11-08T02:48:40.510 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-08T02:48:40.530 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-07_19:34:21-rados-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451192', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-08 01:27:35.654282', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMa3KsPclshNBZmGrUQIMOaXx88aFFEmGaFMAx5bnOxvJQBJWr/IQwg0ynkpohutR/+AV6VXAq8jOV7MWmaFAFg='} 2023-11-08T02:48:40.561 INFO:teuthology.orchestra.console.smithi149:Power off 2023-11-08T02:48:40.562 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-08T02:48:40.587 DEBUG:teuthology.orchestra.console.smithi149:power off output: Chassis Power Control: Down/Off 2023-11-08T02:48:40.588 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T02:48:40.599 INFO:teuthology.orchestra.console.smithi164:Power off 2023-11-08T02:48:40.600 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-08T02:48:40.608 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is off 2023-11-08T02:48:40.621 DEBUG:teuthology.orchestra.console.smithi164:power off output: Chassis Power Control: Down/Off 2023-11-08T02:48:40.621 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T02:48:40.641 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is off 2023-11-08T02:48:40.710 INFO:teuthology.orchestra.console.smithi149:Power off completed 2023-11-08T02:48:40.743 INFO:teuthology.orchestra.console.smithi164:Power off completed 2023-11-08T02:48:40.859 INFO:teuthology.lock.ops:unlocked: smithi149.front.sepia.ceph.com 2023-11-08T02:48:40.885 INFO:teuthology.lock.ops:unlocked: smithi164.front.sepia.ceph.com