2023-08-15T22:03:27.973 INFO:root:teuthology version: 0.0.1.dev183+g5435569 2023-08-15T22:03:27.974 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-15T22:03:32.474 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff316598f70>, signals=[15]) 2023-08-15T22:03:32.475 INFO:teuthology.lock.ops:Start node 'smithi061.front.sepia.ceph.com' reimaging 2023-08-15T22:03:32.476 INFO:teuthology.lock.ops:Updating [smithi061.front.sepia.ceph.com]: reset os type and version on server 2023-08-15T22:03:32.476 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server 2023-08-15T22:03:37.507 INFO:teuthology.lock.ops:Start node 'smithi136.front.sepia.ceph.com' reimaging 2023-08-15T22:03:37.508 INFO:teuthology.lock.ops:Updating [smithi136.front.sepia.ceph.com]: reset os type and version on server 2023-08-15T22:03:37.508 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2023-08-15T22:03:37.510 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /host {"name": "smithi061"} 2023-08-15T22:03:47.538 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /host {"name": "smithi136"} 2023-08-15T22:03:56.270 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-08-15T22:04:08.500 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-08-15T22:04:13.506 DEBUG:teuthology.provision.fog.smithi061:Requesting image smithi_centos_8.stream (ID 3) 2023-08-15T22:04:13.506 DEBUG:teuthology.provision.fog.smithi061:FOG request: PUT /host/166 {"imageID": 3} 2023-08-15T22:04:18.553 DEBUG:teuthology.provision.fog.smithi136:Requesting image smithi_centos_8.stream (ID 3) 2023-08-15T22:04:18.553 DEBUG:teuthology.provision.fog.smithi136:FOG request: PUT /host/241 {"imageID": 3} 2023-08-15T22:04:23.564 INFO:teuthology.provision.fog.smithi061:Scheduling deploy of centos 8.stream 2023-08-15T22:04:23.565 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None 2023-08-15T22:04:27.808 INFO:teuthology.provision.fog.smithi136:Scheduling deploy of centos 8.stream 2023-08-15T22:04:27.808 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None 2023-08-15T22:04:27.831 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /tasktype {"name": "deploy"} 2023-08-15T22:04:27.863 DEBUG:teuthology.provision.fog.smithi061:FOG request: POST /host/166/task {"taskTypeID": 1} 2023-08-15T22:04:27.874 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /tasktype {"name": "deploy"} 2023-08-15T22:04:27.909 DEBUG:teuthology.provision.fog.smithi136:FOG request: POST /host/241/task {"taskTypeID": 1} 2023-08-15T22:04:27.915 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None 2023-08-15T22:04:27.968 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None 2023-08-15T22:04:38.037 INFO:teuthology.orchestra.console.smithi061:Power off 2023-08-15T22:04:38.038 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-15T22:04:38.052 INFO:teuthology.orchestra.console.smithi136:Power off 2023-08-15T22:04:38.053 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-15T22:04:43.063 DEBUG:teuthology.orchestra.console.smithi061:power off output: Chassis Power Control: Down/Off 2023-08-15T22:04:43.064 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:04:43.074 DEBUG:teuthology.orchestra.console.smithi136:power off output: Chassis Power Control: Down/Off 2023-08-15T22:04:43.075 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:04:43.094 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off 2023-08-15T22:04:43.195 INFO:teuthology.orchestra.console.smithi136:Power off completed 2023-08-15T22:04:43.296 INFO:teuthology.orchestra.console.smithi136:Power on 2023-08-15T22:04:43.296 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-08-15T22:04:43.316 DEBUG:teuthology.orchestra.console.smithi136:power on output: Chassis Power Control: Up/On 2023-08-15T22:04:43.316 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:04:43.335 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off 2023-08-15T22:04:44.072 DEBUG:teuthology.orchestra.console.smithi061:check power output: 2023-08-15T22:04:47.340 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:04:48.074 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:04:48.449 DEBUG:teuthology.orchestra.console.smithi136:check power output: 2023-08-15T22:04:49.183 DEBUG:teuthology.orchestra.console.smithi061:check power output: 2023-08-15T22:04:52.450 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:04:53.185 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:04:53.561 DEBUG:teuthology.orchestra.console.smithi136:check power output: 2023-08-15T22:04:54.294 DEBUG:teuthology.orchestra.console.smithi061:check power output: 2023-08-15T22:04:57.565 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:04:58.296 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:04:58.675 DEBUG:teuthology.orchestra.console.smithi136:check power output: 2023-08-15T22:04:59.406 DEBUG:teuthology.orchestra.console.smithi061:check power output: 2023-08-15T22:05:02.679 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:05:03.407 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:05:03.790 DEBUG:teuthology.orchestra.console.smithi136:check power output: 2023-08-15T22:05:04.518 DEBUG:teuthology.orchestra.console.smithi061:check power output: 2023-08-15T22:05:07.794 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:05:08.521 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:05:08.905 DEBUG:teuthology.orchestra.console.smithi136:check power output: 2023-08-15T22:05:09.631 DEBUG:teuthology.orchestra.console.smithi061:check power output: 2023-08-15T22:05:12.906 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:05:13.633 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:05:14.015 DEBUG:teuthology.orchestra.console.smithi136:check power output: 2023-08-15T22:05:14.742 DEBUG:teuthology.orchestra.console.smithi061:check power output: 2023-08-15T22:05:18.016 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:05:18.125 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2023-08-15T22:05:18.226 INFO:teuthology.orchestra.console.smithi136:Power on completed 2023-08-15T22:05:18.326 INFO:teuthology.provision.fog.smithi136:Waiting for deploy to finish 2023-08-15T22:05:18.327 DEBUG:teuthology.provision.fog.smithi136:FOG request: GET /task/active None 2023-08-15T22:05:18.744 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:05:18.788 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-15T22:05:19.854 DEBUG:teuthology.orchestra.console.smithi061:check power output: 2023-08-15T22:05:23.855 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:05:24.964 DEBUG:teuthology.orchestra.console.smithi061:check power output: 2023-08-15T22:05:28.965 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:05:30.075 DEBUG:teuthology.orchestra.console.smithi061:check power output: 2023-08-15T22:05:34.077 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:05:35.186 DEBUG:teuthology.orchestra.console.smithi061:check power output: 2023-08-15T22:05:37.264 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136 2023-08-15T22:05:39.187 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:05:39.297 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is off 2023-08-15T22:05:39.398 INFO:teuthology.orchestra.console.smithi061:Power off completed 2023-08-15T22:05:39.499 INFO:teuthology.orchestra.console.smithi061:Power on 2023-08-15T22:05:39.499 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-08-15T22:05:43.269 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-15T22:05:44.521 DEBUG:teuthology.orchestra.console.smithi061:power on output: Chassis Power Control: Up/On 2023-08-15T22:05:44.522 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:05:45.530 DEBUG:teuthology.orchestra.console.smithi061:check power output: 2023-08-15T22:05:46.320 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136 2023-08-15T22:05:49.534 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:05:49.644 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is on 2023-08-15T22:05:49.745 INFO:teuthology.orchestra.console.smithi061:Power on completed 2023-08-15T22:05:49.845 INFO:teuthology.provision.fog.smithi061:Waiting for deploy to finish 2023-08-15T22:05:49.846 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None 2023-08-15T22:05:54.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-15T22:06:14.948 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None 2023-08-15T22:06:19.954 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136 2023-08-15T22:06:25.961 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-15T22:06:27.279 WARNING:teuthology.provision.fog.smithi136:[Errno None] Unable to connect to port 22 on 172.21.15.136 2023-08-15T22:06:33.286 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-15T22:06:38.289 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None 2023-08-15T22:06:43.362 ERROR:teuthology.orchestra.connection:Error authenticating with smithi136.front.sepia.ceph.com: Authentication failed. 2023-08-15T22:06:43.363 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-08-15T22:06:43.364 DEBUG:teuthology.orchestra.run.smithi136:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-08-15T22:06:43.365 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:06:44.369 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:06:45.373 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:06:46.378 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:06:47.382 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:06:48.387 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:06:49.393 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:06:50.397 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:06:51.402 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:06:52.406 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:06:53.411 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:06:54.417 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:06:55.422 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:06:56.426 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:06:57.431 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:06:58.387 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None 2023-08-15T22:07:03.393 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:04.397 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:05.401 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:06.405 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:07.409 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:08.412 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:09.416 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:10.420 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:11.424 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:12.429 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:13.434 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:14.439 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:15.443 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:16.448 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:17.453 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:18.458 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:18.494 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None 2023-08-15T22:07:20.248 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:21.252 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:22.257 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:23.262 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:24.267 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:25.272 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:26.277 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented') 2023-08-15T22:07:35.347 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None 2023-08-15T22:07:55.440 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None 2023-08-15T22:08:15.517 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None 2023-08-15T22:08:35.606 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None 2023-08-15T22:08:38.252 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-15T22:08:46.320 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-08-15T22:08:52.322 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-15T22:09:00.400 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-08-15T22:09:06.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-15T22:09:09.456 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-08-15T22:09:15.376 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-08-15T22:09:15.378 ERROR:teuthology.provision.fog:[Errno 104] Connection reset by peer on smithi136 2023-08-15T22:09:15.457 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-15T22:09:23.540 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-08-15T22:09:24.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2023-08-15T22:09:24.380 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-15T22:09:24.602 DEBUG:teuthology.orchestra.run.smithi136:> true 2023-08-15T22:09:24.883 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2023-08-15T22:09:24.884 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-08-15T22:09:24.884 DEBUG:teuthology.orchestra.run.smithi136:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-08-15T22:09:29.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-15T22:09:29.905 INFO:teuthology.provision.fog.smithi136:Node is ready 2023-08-15T22:09:29.906 DEBUG:teuthology.orchestra.run.smithi136:> hostname 2023-08-15T22:09:29.929 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136.front.sepia.ceph.com 2023-08-15T22:09:29.929 DEBUG:teuthology.orchestra.run.smithi136:> grep smithi136.front.sepia.ceph.com /etc/hosts 2023-08-15T22:09:29.990 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136 smithi136.front.sepia.ceph.com smithi136 2023-08-15T22:09:29.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo hostname smithi136 2023-08-15T22:09:30.091 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i -e 's/smithi136.front.sepia.ceph.com/smithi136/g' /etc/hosts 2023-08-15T22:09:30.122 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i -e 's/172.21.15.136/172.21.15.136/g' /etc/hosts 2023-08-15T22:09:30.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i -e 's/smithi136.front.sepia.ceph.com/smithi136/g' /etc/hostname 2023-08-15T22:09:30.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo hostname smithi136 2023-08-15T22:09:30.329 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2023-08-15T22:09:30.388 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Stream" 2023-08-15T22:09:30.388 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="8" 2023-08-15T22:09:30.388 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos" 2023-08-15T22:09:30.388 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora" 2023-08-15T22:09:30.388 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="8" 2023-08-15T22:09:30.388 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el8" 2023-08-15T22:09:30.389 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Stream 8" 2023-08-15T22:09:30.389 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2023-08-15T22:09:30.389 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-08-15T22:09:30.389 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://centos.org/" 2023-08-15T22:09:30.389 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-15T22:09:30.389 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-08-15T22:09:30.389 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-08-15T22:09:30.390 INFO:teuthology.provision.fog.smithi136:Deploy complete! 2023-08-15T22:09:30.391 DEBUG:teuthology.parallel:result is None 2023-08-15T22:09:32.596 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-08-15T22:09:38.597 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-15T22:10:01.968 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-08-15T22:10:07.970 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-15T22:10:15.002 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-08-15T22:10:15.003 DEBUG:teuthology.orchestra.run.smithi061:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-08-15T22:10:40.111 INFO:teuthology.provision.fog.smithi061:Node is ready 2023-08-15T22:10:40.112 DEBUG:teuthology.orchestra.run.smithi061:> hostname 2023-08-15T22:10:40.136 INFO:teuthology.orchestra.run.smithi061.stdout:smithi061.front.sepia.ceph.com 2023-08-15T22:10:40.137 DEBUG:teuthology.orchestra.run.smithi061:> grep smithi061.front.sepia.ceph.com /etc/hosts 2023-08-15T22:10:40.198 INFO:teuthology.orchestra.run.smithi061.stdout:172.21.15.61 smithi061.front.sepia.ceph.com smithi061 2023-08-15T22:10:40.199 DEBUG:teuthology.orchestra.run.smithi061:> sudo hostname smithi061 2023-08-15T22:10:40.323 DEBUG:teuthology.orchestra.run.smithi061:> sudo sed -i -e 's/smithi061.front.sepia.ceph.com/smithi061/g' /etc/hosts 2023-08-15T22:10:40.355 DEBUG:teuthology.orchestra.run.smithi061:> sudo sed -i -e 's/172.21.15.61/172.21.15.61/g' /etc/hosts 2023-08-15T22:10:40.426 DEBUG:teuthology.orchestra.run.smithi061:> sudo sed -i -e 's/smithi061.front.sepia.ceph.com/smithi061/g' /etc/hostname 2023-08-15T22:10:40.496 DEBUG:teuthology.orchestra.run.smithi061:> sudo hostname smithi061 2023-08-15T22:10:40.565 DEBUG:teuthology.orchestra.run.smithi061:> cat /etc/os-release 2023-08-15T22:10:40.623 INFO:teuthology.orchestra.run.smithi061.stdout:NAME="CentOS Stream" 2023-08-15T22:10:40.623 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION="8" 2023-08-15T22:10:40.624 INFO:teuthology.orchestra.run.smithi061.stdout:ID="centos" 2023-08-15T22:10:40.624 INFO:teuthology.orchestra.run.smithi061.stdout:ID_LIKE="rhel fedora" 2023-08-15T22:10:40.628 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_ID="8" 2023-08-15T22:10:40.629 INFO:teuthology.orchestra.run.smithi061.stdout:PLATFORM_ID="platform:el8" 2023-08-15T22:10:40.629 INFO:teuthology.orchestra.run.smithi061.stdout:PRETTY_NAME="CentOS Stream 8" 2023-08-15T22:10:40.629 INFO:teuthology.orchestra.run.smithi061.stdout:ANSI_COLOR="0;31" 2023-08-15T22:10:40.629 INFO:teuthology.orchestra.run.smithi061.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-08-15T22:10:40.630 INFO:teuthology.orchestra.run.smithi061.stdout:HOME_URL="https://centos.org/" 2023-08-15T22:10:40.630 INFO:teuthology.orchestra.run.smithi061.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-15T22:10:40.630 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-08-15T22:10:40.631 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-08-15T22:10:40.631 INFO:teuthology.provision.fog.smithi061:Deploy complete! 2023-08-15T22:10:40.632 DEBUG:teuthology.parallel:result is None 2023-08-15T22:11:15.793 INFO:teuthology.lock.ops:Checking smithi061.front.sepia.ceph.com 2023-08-15T22:11:15.794 INFO:teuthology.lock.ops:New key found. Updating... 2023-08-15T22:11:20.821 INFO:teuthology.lock.ops:Checking smithi136.front.sepia.ceph.com 2023-08-15T22:11:20.821 INFO:teuthology.lock.ops:New key found. Updating... 2023-08-15T22:11:25.847 INFO:teuthology.lock.ops:Updating [smithi061.front.sepia.ceph.com]: set os type and version on server 2023-08-15T22:11:25.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-08-15T22:11:25.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-15T22:11:25.916 DEBUG:teuthology.orchestra.run.smithi061:> true 2023-08-15T22:11:26.056 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-08-15T22:11:26.056 DEBUG:teuthology.orchestra.run.smithi061:> uname -m 2023-08-15T22:11:26.114 INFO:teuthology.orchestra.run.smithi061.stdout:x86_64 2023-08-15T22:11:26.115 DEBUG:teuthology.orchestra.run.smithi061:> cat /etc/os-release 2023-08-15T22:11:26.174 INFO:teuthology.orchestra.run.smithi061.stdout:NAME="CentOS Stream" 2023-08-15T22:11:26.174 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION="8" 2023-08-15T22:11:26.174 INFO:teuthology.orchestra.run.smithi061.stdout:ID="centos" 2023-08-15T22:11:26.175 INFO:teuthology.orchestra.run.smithi061.stdout:ID_LIKE="rhel fedora" 2023-08-15T22:11:26.175 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_ID="8" 2023-08-15T22:11:26.175 INFO:teuthology.orchestra.run.smithi061.stdout:PLATFORM_ID="platform:el8" 2023-08-15T22:11:26.175 INFO:teuthology.orchestra.run.smithi061.stdout:PRETTY_NAME="CentOS Stream 8" 2023-08-15T22:11:26.175 INFO:teuthology.orchestra.run.smithi061.stdout:ANSI_COLOR="0;31" 2023-08-15T22:11:26.175 INFO:teuthology.orchestra.run.smithi061.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-08-15T22:11:26.176 INFO:teuthology.orchestra.run.smithi061.stdout:HOME_URL="https://centos.org/" 2023-08-15T22:11:26.176 INFO:teuthology.orchestra.run.smithi061.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-15T22:11:26.176 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-08-15T22:11:26.176 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-08-15T22:11:26.177 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server 2023-08-15T22:11:31.203 INFO:teuthology.lock.ops:Updating [smithi136.front.sepia.ceph.com]: set os type and version on server 2023-08-15T22:11:31.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2023-08-15T22:11:31.204 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-15T22:11:36.275 DEBUG:teuthology.orchestra.run.smithi136:> true 2023-08-15T22:11:36.411 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2023-08-15T22:11:36.412 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2023-08-15T22:11:36.470 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2023-08-15T22:11:36.471 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2023-08-15T22:11:36.529 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Stream" 2023-08-15T22:11:36.529 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="8" 2023-08-15T22:11:36.530 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos" 2023-08-15T22:11:36.530 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora" 2023-08-15T22:11:36.530 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="8" 2023-08-15T22:11:36.531 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el8" 2023-08-15T22:11:36.531 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Stream 8" 2023-08-15T22:11:36.531 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2023-08-15T22:11:36.531 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-08-15T22:11:36.532 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://centos.org/" 2023-08-15T22:11:36.532 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-15T22:11:36.532 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-08-15T22:11:36.532 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-08-15T22:11:36.533 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2023-08-15T22:11:41.569 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-15T22:11:46.668 INFO:teuthology.dispatcher.supervisor:Running job 7368867 2023-08-15T22:11:46.668 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_54355694caed5e99f46f0c9317b1e9bcf8940415/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368867 --name yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi --description fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} -- /home/teuthworker/archive/yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368867/orig.config.yaml 2023-08-15T22:11:46.671 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368867 2023-08-15T22:11:46.672 INFO:teuthology.dispatcher.supervisor:Job PID: 5099 2023-08-15T22:11:46.672 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-08-15T22:13:46.745 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-15T22:15:51.921 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-15T22:17:52.085 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-15T22:19:57.261 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-15T22:22:02.425 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-15T22:24:07.589 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-15T22:26:11.756 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-15T22:28:16.233 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-15T22:30:20.821 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-15T22:32:25.972 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-15T22:34:31.129 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-15T22:36:36.293 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-15T22:36:41.361 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120 2023-08-15T22:36:51.492 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2023-08-15T22:36:51.492 INFO:teuthology.nuke:Checking targets against current locks 2023-08-15T22:36:52.977 DEBUG:teuthology.nuke:shortname: smithi061 2023-08-15T22:36:52.977 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-15T22:36:52.994 DEBUG:teuthology.nuke:shortname: smithi136 2023-08-15T22:36:52.995 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-15T22:36:52.998 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi061.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368867', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-15 22:03:21.885535', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/mcBsKEt5a2QW61mipFAhi19fMqGE3ffNxJRbUOcEwTZGRxyBb8JdCk4nK978kQW0w6eCXifG6g9v/1LVRFU7dYClBxzqfczNCFj2Ffv+R7GUxC6JkLoElDHX1nSK52v2309uXvZP4aSpiEdRwaaS1nGX/y7V9l11Mh4CoF+fvr7GuYirn8tfXAoTyM8pajJlhW+g6LZuEV5Bg9KM5KadrXyWP12KA2SsugvXYwSyg4oK7AAfe0KZuruMq4HVKsFyqDpDZ8GTV8RhxcRJe4oAJuR5yV7UVGZQzemYd3DsjI87QkrXV+ldtBJlEMdl7gdQpL9vvsJ39jeyoorzLVWY31xwriZjWkN+zF8SR2feqOgeu5StIY6DkekbqFlBaLLxIoAO6VQRhYtABmQrK+7d0z2MrRu8HSkXKiA/E+cLPa1Df8fRDDMSqq0NvlzZkx5y9hvi1096djjfONWgdu5E0jufFYxNLpt4wOJLdAwIBHTbIfZqkJPKyDVUSBEeaxk='} 2023-08-15T22:36:53.015 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-15_16:56:50-fs-wip-yuri7-testing-2023-08-15-0741-pacific-distro-default-smithi/7368867', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-15 22:03:21.884006', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8+Na0h/bPQnDAtwa0ZYidJCSrKnGwLWC1pJ3brSwkdhsBNoTsjB5WZ+UkzDhRDgPjVsAsZM8cgIskOQb6nSkphPz04FIeul3vzeRF57QAsKa+WyvJ1apClxrA358KyB3E5a6PJBPgeOWKm6IYC6yobC93v6u+hcPbaXppFNV44RLFIf0/becFgGbknmtEZFb+Q5ed4IW5GK411+W1ResXJsQxPPlig6Cj7N61wvyNdf0p9FaRH6xFuOiWYYyRy1VpfFSMfjpkvv5EvktOQGPv/O0uxnyQ9lUaqVnkWOHiu1BnaIcyKH9a89TKxcJs9s/YUb5v8PoVI2q1HDSgf/Xv1AyoAyZtRUuL/P8e8WdsCFq83ByzphnxjYj5xl9pUsJvRVXi1AR/EhlWLzyq+35SfLvV6hqOJpho4X5c7l2mZY1+fOM1eRZwSVSdNVKZzerTaZmqC9wVGu9kPj4XVO4GfKsY84ZcEWDQLBrdM6TBM58bUw7or4Bp+tcBRr86CdU='} 2023-08-15T22:36:53.044 INFO:teuthology.orchestra.console.smithi061:Power off 2023-08-15T22:36:53.045 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-15T22:36:53.064 INFO:teuthology.orchestra.console.smithi136:Power off 2023-08-15T22:36:53.065 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-15T22:36:53.074 DEBUG:teuthology.orchestra.console.smithi061:power off output: Chassis Power Control: Down/Off 2023-08-15T22:36:53.075 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:36:53.086 DEBUG:teuthology.orchestra.console.smithi136:power off output: Chassis Power Control: Down/Off 2023-08-15T22:36:53.087 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-15T22:36:53.097 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is off 2023-08-15T22:36:53.109 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off 2023-08-15T22:36:53.199 INFO:teuthology.orchestra.console.smithi061:Power off completed 2023-08-15T22:36:53.211 INFO:teuthology.orchestra.console.smithi136:Power off completed 2023-08-15T22:36:53.344 INFO:teuthology.lock.ops:unlocked: smithi061.front.sepia.ceph.com 2023-08-15T22:36:53.353 INFO:teuthology.lock.ops:unlocked: smithi136.front.sepia.ceph.com