2023-07-08T06:31:49.541 INFO:root:teuthology version: 0.0.1.dev140+gb53f236 2023-07-08T06:31:49.543 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-08T06:31:49.543 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-08T06:31:49.668 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f916f9a28b0>, signals=[15]) 2023-07-08T06:31:49.669 INFO:teuthology.lock.ops:Start node 'smithi097.front.sepia.ceph.com' reimaging 2023-07-08T06:31:49.670 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: reset os type and version on server 2023-07-08T06:31:49.670 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2023-07-08T06:31:49.690 INFO:teuthology.lock.ops:Start node 'smithi192.front.sepia.ceph.com' reimaging 2023-07-08T06:31:49.690 INFO:teuthology.lock.ops:Updating [smithi192.front.sepia.ceph.com]: reset os type and version on server 2023-07-08T06:31:49.691 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2023-07-08T06:31:49.692 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /host {"name": "smithi097"} 2023-07-08T06:31:49.715 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /host {"name": "smithi192"} 2023-07-08T06:31:49.762 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-07-08T06:31:49.778 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-07-08T06:31:49.800 DEBUG:teuthology.provision.fog.smithi097:Requesting image smithi_centos_8.stream (ID 3) 2023-07-08T06:31:49.800 DEBUG:teuthology.provision.fog.smithi097:FOG request: PUT /host/202 {"imageID": 3} 2023-07-08T06:31:49.820 DEBUG:teuthology.provision.fog.smithi192:Requesting image smithi_centos_8.stream (ID 3) 2023-07-08T06:31:49.820 DEBUG:teuthology.provision.fog.smithi192:FOG request: PUT /host/297 {"imageID": 3} 2023-07-08T06:31:49.847 INFO:teuthology.provision.fog.smithi097:Scheduling deploy of centos 8.stream 2023-07-08T06:31:49.847 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:31:49.858 INFO:teuthology.provision.fog.smithi192:Scheduling deploy of centos 8.stream 2023-07-08T06:31:49.858 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:31:49.952 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /tasktype {"name": "deploy"} 2023-07-08T06:31:49.962 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /tasktype {"name": "deploy"} 2023-07-08T06:31:49.993 DEBUG:teuthology.provision.fog.smithi097:FOG request: POST /host/202/task {"taskTypeID": 1} 2023-07-08T06:31:49.995 DEBUG:teuthology.provision.fog.smithi192:FOG request: POST /host/297/task {"taskTypeID": 1} 2023-07-08T06:31:50.117 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:31:50.131 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:31:50.270 INFO:teuthology.orchestra.console:Power off smithi097 2023-07-08T06:31:50.271 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-08T06:31:50.283 INFO:teuthology.orchestra.console:Power off smithi192 2023-07-08T06:31:50.284 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-08T06:31:50.291 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-08T06:31:50.303 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-08T06:31:50.412 INFO:teuthology.orchestra.console:Power off for smithi097 completed 2023-07-08T06:31:50.424 INFO:teuthology.orchestra.console:Power off for smithi192 completed 2023-07-08T06:31:50.514 INFO:teuthology.orchestra.console:Power on smithi097 2023-07-08T06:31:50.514 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-07-08T06:31:50.525 INFO:teuthology.orchestra.console:Power on smithi192 2023-07-08T06:31:50.525 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-07-08T06:31:50.534 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-08T06:31:50.546 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-08T06:31:54.560 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-08T06:31:54.569 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-08T06:31:58.676 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-08T06:31:58.683 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-08T06:31:58.900 INFO:teuthology.orchestra.console:Power on for smithi192 completed 2023-07-08T06:31:59.001 INFO:teuthology.provision.fog.smithi192:Waiting for deploy to finish 2023-07-08T06:31:59.002 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:31:59.003 INFO:teuthology.orchestra.console:Power on for smithi097 completed 2023-07-08T06:31:59.104 INFO:teuthology.provision.fog.smithi097:Waiting for deploy to finish 2023-07-08T06:31:59.104 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:32:14.221 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:32:14.321 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:32:29.456 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:32:29.548 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:32:44.699 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:32:44.781 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:32:59.949 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:33:00.033 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:33:15.209 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:33:15.301 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:33:30.526 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:33:30.601 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:33:45.860 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:33:45.934 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:34:01.276 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:34:01.277 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:34:18.784 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:34:18.785 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:34:35.318 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:34:35.524 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:34:51.504 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:34:51.505 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:35:08.862 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:35:08.904 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:35:26.389 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:35:26.392 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:35:43.528 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:35:43.703 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:36:00.888 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:36:01.189 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:36:17.815 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:36:17.889 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:36:34.928 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:36:34.967 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:36:50.695 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:36:50.767 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:37:06.704 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:37:06.804 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:37:22.824 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:37:22.957 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:37:39.476 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:37:39.491 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:37:55.168 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:37:55.202 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:38:10.519 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:38:10.521 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:38:25.879 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:38:25.883 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:38:41.260 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:38:41.261 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:38:56.584 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:38:56.586 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:39:11.812 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:39:11.827 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:39:27.261 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:39:27.263 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:39:42.615 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:39:42.617 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:39:57.984 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:39:57.985 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:40:13.355 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:40:13.386 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:40:28.780 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:40:28.781 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:40:44.089 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:40:44.115 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:40:59.448 DEBUG:teuthology.provision.fog.smithi192:FOG request: GET /task/active None 2023-07-08T06:40:59.449 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:40:59.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:41:02.864 WARNING:teuthology.provision.fog.smithi192:[Errno None] Unable to connect to port 22 on 172.21.15.192 2023-07-08T06:41:08.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:41:11.952 WARNING:teuthology.provision.fog.smithi192:[Errno None] Unable to connect to port 22 on 172.21.15.192 2023-07-08T06:41:14.794 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:41:17.956 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:41:21.008 WARNING:teuthology.provision.fog.smithi192:[Errno None] Unable to connect to port 22 on 172.21.15.192 2023-07-08T06:41:27.012 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:41:30.064 WARNING:teuthology.provision.fog.smithi192:[Errno None] Unable to connect to port 22 on 172.21.15.192 2023-07-08T06:41:30.085 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:41:36.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:41:39.152 WARNING:teuthology.provision.fog.smithi192:[Errno None] Unable to connect to port 22 on 172.21.15.192 2023-07-08T06:41:45.159 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:41:45.353 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:41:48.240 WARNING:teuthology.provision.fog.smithi192:[Errno None] Unable to connect to port 22 on 172.21.15.192 2023-07-08T06:41:54.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:41:57.296 WARNING:teuthology.provision.fog.smithi192:[Errno None] Unable to connect to port 22 on 172.21.15.192 2023-07-08T06:42:00.608 DEBUG:teuthology.provision.fog.smithi097:FOG request: GET /task/active None 2023-07-08T06:42:00.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:42:03.300 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:42:03.860 WARNING:teuthology.provision.fog.smithi097:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-07-08T06:42:09.864 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:42:12.944 WARNING:teuthology.provision.fog.smithi097:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-07-08T06:42:18.948 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:42:21.680 WARNING:teuthology.provision.fog.smithi192:[Errno None] Unable to connect to port 22 on 172.21.15.192 2023-07-08T06:42:22.000 WARNING:teuthology.provision.fog.smithi097:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-07-08T06:42:27.684 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:42:28.002 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:42:30.736 WARNING:teuthology.provision.fog.smithi192:[Errno None] Unable to connect to port 22 on 172.21.15.192 2023-07-08T06:42:31.056 WARNING:teuthology.provision.fog.smithi097:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-07-08T06:42:36.740 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:42:37.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:42:37.688 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-07-08T06:42:37.688 DEBUG:teuthology.orchestra.run.smithi192:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-07-08T06:42:40.112 WARNING:teuthology.provision.fog.smithi097:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-07-08T06:42:46.114 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:42:49.168 WARNING:teuthology.provision.fog.smithi097:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-07-08T06:42:55.170 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:42:58.228 WARNING:teuthology.provision.fog.smithi097:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-07-08T06:43:03.196 INFO:teuthology.provision.fog.smithi192:Node is ready 2023-07-08T06:43:03.197 DEBUG:teuthology.orchestra.run.smithi192:> hostname 2023-07-08T06:43:03.220 INFO:teuthology.orchestra.run.smithi192.stdout:smithi192.front.sepia.ceph.com 2023-07-08T06:43:03.221 DEBUG:teuthology.orchestra.run.smithi192:> grep smithi192.front.sepia.ceph.com /etc/hosts 2023-07-08T06:43:03.280 INFO:teuthology.orchestra.run.smithi192.stdout:172.21.15.192 smithi192.front.sepia.ceph.com smithi192 2023-07-08T06:43:03.281 DEBUG:teuthology.orchestra.run.smithi192:> sudo hostname smithi192 2023-07-08T06:43:03.425 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i -e 's/smithi192.front.sepia.ceph.com/smithi192/g' /etc/hosts 2023-07-08T06:43:03.456 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i -e 's/172.21.15.192/172.21.15.192/g' /etc/hosts 2023-07-08T06:43:03.527 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i -e 's/smithi192.front.sepia.ceph.com/smithi192/g' /etc/hostname 2023-07-08T06:43:03.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo hostname smithi192 2023-07-08T06:43:03.666 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2023-07-08T06:43:03.725 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Stream" 2023-07-08T06:43:03.725 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="8" 2023-07-08T06:43:03.726 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2023-07-08T06:43:03.726 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2023-07-08T06:43:03.726 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="8" 2023-07-08T06:43:03.726 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el8" 2023-07-08T06:43:03.726 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Stream 8" 2023-07-08T06:43:03.726 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2023-07-08T06:43:03.726 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-07-08T06:43:03.727 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/" 2023-07-08T06:43:03.727 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-08T06:43:03.727 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-08T06:43:03.727 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-07-08T06:43:03.728 INFO:teuthology.provision.fog.smithi192:Deploy complete! 2023-07-08T06:43:03.729 DEBUG:teuthology.parallel:result is None 2023-07-08T06:43:04.230 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:43:22.608 WARNING:teuthology.provision.fog.smithi097:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-07-08T06:43:28.612 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:43:31.664 WARNING:teuthology.provision.fog.smithi097:[Errno None] Unable to connect to port 22 on 172.21.15.97 2023-07-08T06:43:37.669 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:43:40.755 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-07-08T06:43:40.756 DEBUG:teuthology.orchestra.run.smithi097:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-07-08T06:44:06.283 INFO:teuthology.provision.fog.smithi097:Node is ready 2023-07-08T06:44:06.283 DEBUG:teuthology.orchestra.run.smithi097:> hostname 2023-07-08T06:44:06.308 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097.front.sepia.ceph.com 2023-07-08T06:44:06.308 DEBUG:teuthology.orchestra.run.smithi097:> grep smithi097.front.sepia.ceph.com /etc/hosts 2023-07-08T06:44:06.369 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97 smithi097.front.sepia.ceph.com smithi097 2023-07-08T06:44:06.369 DEBUG:teuthology.orchestra.run.smithi097:> sudo hostname smithi097 2023-07-08T06:44:06.489 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i -e 's/smithi097.front.sepia.ceph.com/smithi097/g' /etc/hosts 2023-07-08T06:44:06.520 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i -e 's/172.21.15.97/172.21.15.97/g' /etc/hosts 2023-07-08T06:44:06.590 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i -e 's/smithi097.front.sepia.ceph.com/smithi097/g' /etc/hostname 2023-07-08T06:44:06.660 DEBUG:teuthology.orchestra.run.smithi097:> sudo hostname smithi097 2023-07-08T06:44:06.728 DEBUG:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2023-07-08T06:44:06.786 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Stream" 2023-07-08T06:44:06.786 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="8" 2023-07-08T06:44:06.786 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2023-07-08T06:44:06.787 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2023-07-08T06:44:06.787 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="8" 2023-07-08T06:44:06.787 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el8" 2023-07-08T06:44:06.787 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Stream 8" 2023-07-08T06:44:06.787 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2023-07-08T06:44:06.787 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-07-08T06:44:06.787 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://centos.org/" 2023-07-08T06:44:06.788 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-08T06:44:06.788 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-08T06:44:06.788 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-07-08T06:44:06.788 INFO:teuthology.provision.fog.smithi097:Deploy complete! 2023-07-08T06:44:06.788 DEBUG:teuthology.parallel:result is None 2023-07-08T06:44:06.944 INFO:teuthology.lock.ops:Checking smithi097.front.sepia.ceph.com 2023-07-08T06:44:06.945 INFO:teuthology.lock.ops:New key found. Updating... 2023-07-08T06:44:06.967 INFO:teuthology.lock.ops:Checking smithi192.front.sepia.ceph.com 2023-07-08T06:44:06.967 INFO:teuthology.lock.ops:New key found. Updating... 2023-07-08T06:44:06.986 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: set os type and version on server 2023-07-08T06:44:06.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-07-08T06:44:06.989 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:44:07.053 DEBUG:teuthology.orchestra.run.smithi097:> true 2023-07-08T06:44:07.122 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-07-08T06:44:07.123 DEBUG:teuthology.orchestra.run.smithi097:> uname -m 2023-07-08T06:44:07.181 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2023-07-08T06:44:07.181 DEBUG:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2023-07-08T06:44:07.239 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Stream" 2023-07-08T06:44:07.239 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="8" 2023-07-08T06:44:07.239 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2023-07-08T06:44:07.239 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2023-07-08T06:44:07.239 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="8" 2023-07-08T06:44:07.240 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el8" 2023-07-08T06:44:07.240 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Stream 8" 2023-07-08T06:44:07.240 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2023-07-08T06:44:07.240 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-07-08T06:44:07.240 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://centos.org/" 2023-07-08T06:44:07.240 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-08T06:44:07.241 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-08T06:44:07.241 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-07-08T06:44:07.241 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2023-07-08T06:44:07.268 INFO:teuthology.lock.ops:Updating [smithi192.front.sepia.ceph.com]: set os type and version on server 2023-07-08T06:44:07.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-07-08T06:44:07.269 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-08T06:44:07.337 DEBUG:teuthology.orchestra.run.smithi192:> true 2023-07-08T06:44:07.479 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-07-08T06:44:07.479 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2023-07-08T06:44:07.539 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2023-07-08T06:44:07.539 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2023-07-08T06:44:07.599 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Stream" 2023-07-08T06:44:07.599 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="8" 2023-07-08T06:44:07.599 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2023-07-08T06:44:07.599 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2023-07-08T06:44:07.599 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="8" 2023-07-08T06:44:07.600 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el8" 2023-07-08T06:44:07.600 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Stream 8" 2023-07-08T06:44:07.600 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2023-07-08T06:44:07.600 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-07-08T06:44:07.600 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/" 2023-07-08T06:44:07.600 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-08T06:44:07.600 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-08T06:44:07.601 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-07-08T06:44:07.601 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2023-07-08T06:44:07.626 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-08T06:44:07.626 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-08T06:44:07.729 INFO:teuthology.dispatcher.supervisor:Running job 7328950 2023-07-08T06:44:07.730 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_b53f236c1077f1d295baafb9e0ca9a4b447019d7/virtualenv/bin/teuthology -v --owner scheduled_adking@teuthology --archive /home/teuthworker/archive/adking-2023-07-08_06:30:43-orch:cephadm-wip-adk4-testing-2023-07-07-2139-distro-default-smithi/7328950 --name adking-2023-07-08_06:30:43-orch:cephadm-wip-adk4-testing-2023-07-07-2139-distro-default-smithi --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/adking-2023-07-08_06:30:43-orch:cephadm-wip-adk4-testing-2023-07-07-2139-distro-default-smithi/7328950/orig.config.yaml 2023-07-08T06:44:07.733 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/adking-2023-07-08_06:30:43-orch:cephadm-wip-adk4-testing-2023-07-07-2139-distro-default-smithi/7328950 2023-07-08T06:44:07.734 INFO:teuthology.dispatcher.supervisor:Job PID: 27799 2023-07-08T06:44:07.734 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-07-08T06:46:07.834 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-08T06:46:07.835 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-08T06:48:08.014 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-08T06:48:08.015 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-08T06:50:08.227 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-08T06:50:08.228 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-08T06:52:08.460 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-08T06:52:08.461 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-08T06:54:08.673 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-08T06:54:08.674 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-08T06:56:08.888 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-08T06:56:08.888 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-08T06:58:09.073 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-08T06:58:09.074 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-08T07:00:09.245 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-08T07:00:09.246 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-08T07:02:09.439 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-08T07:02:09.439 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-08T07:04:09.609 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-08T07:04:09.609 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-08T07:06:09.789 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-08T07:06:09.789 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-08T07:06:09.863 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120 2023-07-08T07:06:10.009 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2023-07-08T07:06:10.010 INFO:teuthology.nuke:Checking targets against current locks 2023-07-08T07:06:10.060 DEBUG:teuthology.nuke:shortname: smithi097 2023-07-08T07:06:10.061 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-08T07:06:10.090 DEBUG:teuthology.nuke:shortname: smithi192 2023-07-08T07:06:10.091 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-08T07:06:10.095 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2023-07-08_06:30:43-orch:cephadm-wip-adk4-testing-2023-07-07-2139-distro-default-smithi/7328950', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-08 06:31:48.738490', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZGFl1N8NM47NM2cRZ8zyvZ6RrxqTjd+Mnv0YT7R/ClK/iakeiM5RUNFUkjaJt0scVMZaClxjC3l63hP1JXtdZBo22YJ7fw3vIGE88dw5rPb5uz5o1PckEaMtSwjneAmf04kZM25umjyaWsYq+heq+owOvuHB98z5JSC3y3hsm/Uh61qeG7RSgvoo/eWfN0eXzQCFbWu7WgYqa7n4VvBbD7xrJnT6m7B2YZLqZ6W/ZHgoWEkDjtGpcbfSlgcqzKH8GeohwkQJ0UUAREhZCOvd+PNQyI8U5UBhNOeoB37QkhoJgbhCtNfj9TXLQsWajCa4PHL75wsh3v1B6A0pQ9G/yDxqwZ6ngXPIa9LC3y7udfpQhtOCoxZ6iM6ea4/lTYyf64Bd9BAXmvpINe5Uhiz8SKF6dw2w+D236+TEMlm7j9/twOSFZrxTJjE1VBNlFNY7HWv4vtXoAAdjm/FBihwJ4JZMrQxwlIyRICOAxdyeCL8agXNcidHg6kQyiNLYk6M0='} 2023-07-08T07:06:10.119 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2023-07-08_06:30:43-orch:cephadm-wip-adk4-testing-2023-07-07-2139-distro-default-smithi/7328950', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-08 06:31:48.740077', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBU6EIpPdc7CIWM3/Bc9SuhhdFaQvP21zIsMYPK32Ktn8vsbclVSQvwEBzs6+Zz/k+ndT5wHE4U2ZjN6wZbzQhc='} 2023-07-08T07:06:10.154 INFO:teuthology.orchestra.console:Power off smithi097 2023-07-08T07:06:10.155 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-08T07:06:10.180 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-08T07:06:10.197 INFO:teuthology.orchestra.console:Power off smithi192 2023-07-08T07:06:10.198 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-08T07:06:10.224 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-08T07:06:10.308 INFO:teuthology.orchestra.console:Power off for smithi097 completed 2023-07-08T07:06:10.350 INFO:teuthology.orchestra.console:Power off for smithi192 completed 2023-07-08T07:06:10.471 INFO:teuthology.lock.ops:unlocked: smithi097.front.sepia.ceph.com 2023-07-08T07:06:10.503 INFO:teuthology.lock.ops:unlocked: smithi192.front.sepia.ceph.com