2023-06-22T16:16:03.645 INFO:root:teuthology version: 0.0.1.dev128+g076bbeb 2023-06-22T16:16:03.646 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T16:16:03.647 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-22T16:16:23.810 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc061d328b0>, signals=[15]) 2023-06-22T16:16:23.811 INFO:teuthology.lock.ops:Start node 'smithi152.front.sepia.ceph.com' reimaging 2023-06-22T16:16:23.812 INFO:teuthology.lock.ops:Updating [smithi152.front.sepia.ceph.com]: reset os type and version on server 2023-06-22T16:16:23.812 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2023-06-22T16:16:28.839 INFO:teuthology.lock.ops:Start node 'smithi195.front.sepia.ceph.com' reimaging 2023-06-22T16:16:28.840 INFO:teuthology.lock.ops:Updating [smithi195.front.sepia.ceph.com]: reset os type and version on server 2023-06-22T16:16:28.840 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2023-06-22T16:16:28.841 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /host {"name": "smithi152"} 2023-06-22T16:16:33.172 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /host {"name": "smithi195"} 2023-06-22T16:16:38.224 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-06-22T16:16:38.227 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-06-22T16:16:38.270 DEBUG:teuthology.provision.fog.smithi195:Requesting image smithi_centos_8.stream (ID 3) 2023-06-22T16:16:38.270 DEBUG:teuthology.provision.fog.smithi195:FOG request: PUT /host/300 {"imageID": 3} 2023-06-22T16:16:38.272 DEBUG:teuthology.provision.fog.smithi152:Requesting image smithi_centos_8.stream (ID 3) 2023-06-22T16:16:38.273 DEBUG:teuthology.provision.fog.smithi152:FOG request: PUT /host/257 {"imageID": 3} 2023-06-22T16:16:38.319 INFO:teuthology.provision.fog.smithi152:Scheduling deploy of centos 8.stream 2023-06-22T16:16:38.319 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-06-22T16:16:38.322 INFO:teuthology.provision.fog.smithi195:Scheduling deploy of centos 8.stream 2023-06-22T16:16:38.322 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-22T16:16:38.364 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /tasktype {"name": "deploy"} 2023-06-22T16:16:38.366 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /tasktype {"name": "deploy"} 2023-06-22T16:16:38.398 DEBUG:teuthology.provision.fog.smithi195:FOG request: POST /host/300/task {"taskTypeID": 1} 2023-06-22T16:16:38.401 DEBUG:teuthology.provision.fog.smithi152:FOG request: POST /host/257/task {"taskTypeID": 1} 2023-06-22T16:16:38.469 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-22T16:16:38.470 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-06-22T16:16:38.553 INFO:teuthology.orchestra.console:Power off smithi152 2023-06-22T16:16:38.554 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-22T16:16:38.568 INFO:teuthology.orchestra.console:Power off smithi195 2023-06-22T16:16:38.568 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-22T16:16:38.577 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T16:16:38.589 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T16:16:38.699 INFO:teuthology.orchestra.console:Power off for smithi152 completed 2023-06-22T16:16:38.800 INFO:teuthology.orchestra.console:Power on smithi152 2023-06-22T16:16:38.800 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-22T16:16:38.820 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T16:16:42.612 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T16:16:42.846 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T16:16:46.724 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T16:16:46.957 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T16:16:51.838 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T16:16:52.067 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T16:16:56.949 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T16:16:57.184 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T16:16:57.192 INFO:teuthology.orchestra.console:Power off for smithi195 completed 2023-06-22T16:16:57.294 INFO:teuthology.orchestra.console:Power on smithi195 2023-06-22T16:16:57.294 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-06-22T16:17:02.303 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T16:17:02.320 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T16:17:07.330 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T16:17:07.412 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T16:17:07.542 INFO:teuthology.orchestra.console:Power on for smithi195 completed 2023-06-22T16:17:07.643 INFO:teuthology.provision.fog.smithi195:Waiting for deploy to finish 2023-06-22T16:17:07.643 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-22T16:17:12.525 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T16:17:12.736 INFO:teuthology.orchestra.console:Power on for smithi152 completed 2023-06-22T16:17:12.837 INFO:teuthology.provision.fog.smithi152:Waiting for deploy to finish 2023-06-22T16:17:12.838 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-06-22T16:17:22.696 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-22T16:17:32.896 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-06-22T16:17:42.760 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-22T16:17:47.956 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-06-22T16:17:57.824 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-22T16:18:08.036 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-06-22T16:18:17.888 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-22T16:18:23.092 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-06-22T16:18:32.947 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-22T16:18:38.145 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-06-22T16:18:53.008 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-22T16:18:58.199 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-06-22T16:19:13.080 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-22T16:19:18.089 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-06-22T16:19:38.168 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-06-22T16:19:38.169 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-22T16:19:53.240 DEBUG:teuthology.provision.fog.smithi195:FOG request: GET /task/active None 2023-06-22T16:19:53.241 DEBUG:teuthology.provision.fog.smithi152:FOG request: GET /task/active None 2023-06-22T16:19:53.287 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T16:19:53.290 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T16:20:11.760 WARNING:teuthology.provision.fog.smithi152:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-06-22T16:20:11.761 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195 2023-06-22T16:20:17.764 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T16:20:17.766 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T16:20:20.816 WARNING:teuthology.provision.fog.smithi152:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-06-22T16:20:20.817 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195 2023-06-22T16:20:26.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T16:20:26.824 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T16:20:29.904 WARNING:teuthology.provision.fog.smithi152:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-06-22T16:20:29.905 WARNING:teuthology.provision.fog.smithi195:[Errno None] Unable to connect to port 22 on 172.21.15.195 2023-06-22T16:20:35.908 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T16:20:35.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T16:20:38.960 WARNING:teuthology.provision.fog.smithi152:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-06-22T16:20:44.964 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T16:20:48.016 WARNING:teuthology.provision.fog.smithi152:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-06-22T16:20:54.020 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T16:20:57.076 WARNING:teuthology.provision.fog.smithi152:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-06-22T16:21:03.080 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T16:21:21.648 WARNING:teuthology.provision.fog.smithi152:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-06-22T16:21:27.652 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T16:21:30.704 WARNING:teuthology.provision.fog.smithi152:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-06-22T16:21:35.915 WARNING:teuthology.provision.fog.smithi195:timed out 2023-06-22T16:21:36.707 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T16:21:38.927 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-22T16:21:38.928 DEBUG:teuthology.orchestra.run.smithi152:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-22T16:21:41.917 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T16:21:42.134 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-06-22T16:21:42.135 DEBUG:teuthology.orchestra.run.smithi195:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-06-22T16:21:42.408 INFO:teuthology.provision.fog.smithi195:Node is ready 2023-06-22T16:21:42.408 DEBUG:teuthology.orchestra.run.smithi195:> hostname 2023-06-22T16:21:42.427 INFO:teuthology.orchestra.run.smithi195.stdout:smithi195.front.sepia.ceph.com 2023-06-22T16:21:42.428 DEBUG:teuthology.orchestra.run.smithi195:> grep smithi195.front.sepia.ceph.com /etc/hosts 2023-06-22T16:21:42.487 INFO:teuthology.orchestra.run.smithi195.stdout:172.21.15.195 smithi195.front.sepia.ceph.com smithi195 2023-06-22T16:21:42.487 DEBUG:teuthology.orchestra.run.smithi195:> sudo hostname smithi195 2023-06-22T16:21:42.596 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i -e 's/smithi195.front.sepia.ceph.com/smithi195/g' /etc/hosts 2023-06-22T16:21:42.626 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i -e 's/172.21.15.195/172.21.15.195/g' /etc/hosts 2023-06-22T16:21:42.695 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i -e 's/smithi195.front.sepia.ceph.com/smithi195/g' /etc/hostname 2023-06-22T16:21:42.764 DEBUG:teuthology.orchestra.run.smithi195:> sudo hostname smithi195 2023-06-22T16:21:42.832 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2023-06-22T16:21:42.889 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream" 2023-06-22T16:21:42.889 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="8" 2023-06-22T16:21:42.889 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos" 2023-06-22T16:21:42.890 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora" 2023-06-22T16:21:42.890 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="8" 2023-06-22T16:21:42.890 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el8" 2023-06-22T16:21:42.890 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 8" 2023-06-22T16:21:42.890 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2023-06-22T16:21:42.890 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-06-22T16:21:42.890 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/" 2023-06-22T16:21:42.891 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-22T16:21:42.891 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-22T16:21:42.891 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-06-22T16:21:42.892 INFO:teuthology.provision.fog.smithi195:Deploy complete! 2023-06-22T16:21:42.893 DEBUG:teuthology.parallel:result is None 2023-06-22T16:22:04.051 INFO:teuthology.provision.fog.smithi152:Node is ready 2023-06-22T16:22:04.052 DEBUG:teuthology.orchestra.run.smithi152:> hostname 2023-06-22T16:22:04.075 INFO:teuthology.orchestra.run.smithi152.stdout:smithi152.front.sepia.ceph.com 2023-06-22T16:22:04.076 DEBUG:teuthology.orchestra.run.smithi152:> grep smithi152.front.sepia.ceph.com /etc/hosts 2023-06-22T16:22:04.137 INFO:teuthology.orchestra.run.smithi152.stdout:172.21.15.152 smithi152.front.sepia.ceph.com smithi152 2023-06-22T16:22:04.137 DEBUG:teuthology.orchestra.run.smithi152:> sudo hostname smithi152 2023-06-22T16:22:04.268 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i -e 's/smithi152.front.sepia.ceph.com/smithi152/g' /etc/hosts 2023-06-22T16:22:04.301 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i -e 's/172.21.15.152/172.21.15.152/g' /etc/hosts 2023-06-22T16:22:04.370 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i -e 's/smithi152.front.sepia.ceph.com/smithi152/g' /etc/hostname 2023-06-22T16:22:04.439 DEBUG:teuthology.orchestra.run.smithi152:> sudo hostname smithi152 2023-06-22T16:22:04.507 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2023-06-22T16:22:04.565 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2023-06-22T16:22:04.566 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="8" 2023-06-22T16:22:04.566 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2023-06-22T16:22:04.566 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2023-06-22T16:22:04.566 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="8" 2023-06-22T16:22:04.566 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el8" 2023-06-22T16:22:04.566 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 8" 2023-06-22T16:22:04.567 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2023-06-22T16:22:04.567 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-06-22T16:22:04.567 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2023-06-22T16:22:04.567 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-22T16:22:04.567 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-22T16:22:04.567 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-06-22T16:22:04.568 INFO:teuthology.provision.fog.smithi152:Deploy complete! 2023-06-22T16:22:04.568 DEBUG:teuthology.parallel:result is None 2023-06-22T16:22:04.719 INFO:teuthology.lock.ops:Checking smithi152.front.sepia.ceph.com 2023-06-22T16:22:04.720 INFO:teuthology.lock.ops:New key found. Updating... 2023-06-22T16:22:04.745 INFO:teuthology.lock.ops:Checking smithi195.front.sepia.ceph.com 2023-06-22T16:22:04.745 INFO:teuthology.lock.ops:New key found. Updating... 2023-06-22T16:22:04.772 INFO:teuthology.lock.ops:Updating [smithi152.front.sepia.ceph.com]: set os type and version on server 2023-06-22T16:22:04.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2023-06-22T16:22:04.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T16:22:04.839 DEBUG:teuthology.orchestra.run.smithi152:> true 2023-06-22T16:22:04.906 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2023-06-22T16:22:04.963 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2023-06-22T16:22:04.964 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2023-06-22T16:22:05.021 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2023-06-22T16:22:05.021 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="8" 2023-06-22T16:22:05.021 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2023-06-22T16:22:05.021 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2023-06-22T16:22:05.022 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="8" 2023-06-22T16:22:05.022 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el8" 2023-06-22T16:22:05.022 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 8" 2023-06-22T16:22:05.022 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2023-06-22T16:22:05.022 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-06-22T16:22:05.022 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2023-06-22T16:22:05.022 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-22T16:22:05.022 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-22T16:22:05.023 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-06-22T16:22:05.023 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2023-06-22T16:22:05.045 INFO:teuthology.lock.ops:Updating [smithi195.front.sepia.ceph.com]: set os type and version on server 2023-06-22T16:22:05.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2023-06-22T16:22:05.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-22T16:22:05.111 DEBUG:teuthology.orchestra.run.smithi195:> true 2023-06-22T16:22:05.251 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2023-06-22T16:22:05.309 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2023-06-22T16:22:05.309 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2023-06-22T16:22:05.368 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream" 2023-06-22T16:22:05.368 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="8" 2023-06-22T16:22:05.368 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos" 2023-06-22T16:22:05.368 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora" 2023-06-22T16:22:05.368 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="8" 2023-06-22T16:22:05.369 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el8" 2023-06-22T16:22:05.369 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 8" 2023-06-22T16:22:05.369 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2023-06-22T16:22:05.369 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-06-22T16:22:05.369 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/" 2023-06-22T16:22:05.369 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-22T16:22:05.369 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-22T16:22:05.370 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-06-22T16:22:05.370 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2023-06-22T16:22:05.395 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T16:22:05.396 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-22T16:22:05.502 INFO:teuthology.dispatcher.supervisor:Running job 7311799 2023-06-22T16:22:05.503 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311799 --name yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311799/orig.config.yaml 2023-06-22T16:22:05.506 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311799 2023-06-22T16:22:05.507 INFO:teuthology.dispatcher.supervisor:Job PID: 1857 2023-06-22T16:22:05.508 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-06-22T16:24:05.608 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T16:24:05.609 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-22T16:26:05.773 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T16:26:05.774 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-22T16:28:05.949 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T16:28:05.949 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-22T16:30:06.126 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T16:30:06.128 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-22T16:32:06.301 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T16:32:06.302 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-22T16:34:06.476 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T16:34:06.477 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-22T16:36:06.653 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T16:36:06.679 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-22T16:38:06.853 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T16:38:06.854 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-22T16:40:07.000 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T16:40:07.001 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-22T16:42:07.180 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T16:42:07.181 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-22T16:44:07.369 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T16:44:07.370 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-22T16:46:07.549 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T16:46:07.550 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-22T16:48:07.726 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-22T16:48:07.727 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-06-22T16:48:07.790 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120 2023-06-22T16:48:07.981 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2023-06-22T16:48:07.982 INFO:teuthology.nuke:Checking targets against current locks 2023-06-22T16:48:08.035 DEBUG:teuthology.nuke:shortname: smithi152 2023-06-22T16:48:08.036 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-22T16:48:08.053 DEBUG:teuthology.nuke:shortname: smithi195 2023-06-22T16:48:08.054 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-22T16:48:08.062 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311799', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-22 16:16:02.781503', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDR6gGWQAV7vsFS44Kvo77KgDmlECX1ja2Y5NbS/1G+FCzmJOhAiA9QKXvZQIVYaeX3ntZO0VvWYgd+4iu2xdzVSjHr1RBWLscG787ydHuYOYfwUshOqXChZ3NUCKiD1o14N5Uq2g4AJHddIq8ZKI8OucFZwvDIRSJeqRPL9kNxIhw+R6oeU8ColEZGLieBfpWde8fRPCzliyHwM4XlGZ2YJxC4lA65Ak7nR4ZJ0Ye3ojg6G8RGYNpWVHo7KMZgRSr4KT6YMwlzeYc2KDdEw6AUqQ/SCOJD+qWzLb3w5Am7+u9t9qtAzaiiFXnwcafdtpXd3HIu70dy4zxcVIhUTrXG5YzCdIJWSoEx7UnAeAztlrUZb3S0WbFNy0xOFgAVWwCdZwrz1ZTOcd4vupLba0ZAxhKYEZLzJDkF+ZpPiscJZqswJ1rkjsPrQvNVfGt5R2ZHhKBEMfafdMahN37WPZhnauGgg7FVbYxa5Q0JcGp8mkk2Yfe+afBVAOFp4kWl21s='} 2023-06-22T16:48:08.074 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311799', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-22 16:16:02.783826', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDxUipujqoGoZ2g2JJX/Jg5SMBTlR6r26vYnou9OzYhUZ2a03c7t1jAE6JiCvTxGeQeqcApLB9qB9AP6GIepxyY='} 2023-06-22T16:48:08.113 INFO:teuthology.orchestra.console:Power off smithi152 2023-06-22T16:48:08.114 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-22T16:48:08.128 INFO:teuthology.orchestra.console:Power off smithi195 2023-06-22T16:48:08.129 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-22T16:48:08.139 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T16:48:08.150 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-22T16:48:08.261 INFO:teuthology.orchestra.console:Power off for smithi152 completed 2023-06-22T16:48:08.272 INFO:teuthology.orchestra.console:Power off for smithi195 completed 2023-06-22T16:48:08.422 INFO:teuthology.lock.ops:unlocked: smithi152.front.sepia.ceph.com 2023-06-22T16:48:08.431 INFO:teuthology.lock.ops:unlocked: smithi195.front.sepia.ceph.com