2023-11-02T15:25:05.212 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2
2023-11-02T15:25:05.213 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-02T15:25:05.338 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fd9e6e6cb50>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fd9e7957670>, signals=[15])
2023-11-02T15:25:05.340 INFO:teuthology.lock.ops:Start node 'smithi088.front.sepia.ceph.com' reimaging
2023-11-02T15:25:05.340 INFO:teuthology.lock.ops:Updating [smithi088.front.sepia.ceph.com]: reset os type and version on server
2023-11-02T15:25:05.340 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server
2023-11-02T15:25:05.356 INFO:teuthology.lock.ops:Start node 'smithi150.front.sepia.ceph.com' reimaging
2023-11-02T15:25:05.356 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: reset os type and version on server
2023-11-02T15:25:05.356 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server
2023-11-02T15:25:05.358 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /host {"name": "smithi088"}
2023-11-02T15:25:05.372 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /host {"name": "smithi150"}
2023-11-02T15:25:05.428 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-11-02T15:25:05.435 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /image {"name": "smithi_centos_8.stream"}
2023-11-02T15:25:05.472 DEBUG:teuthology.provision.fog.smithi088:Requesting image smithi_centos_8.stream (ID 3)
2023-11-02T15:25:05.472 DEBUG:teuthology.provision.fog.smithi088:FOG request: PUT /host/193 {"imageID": 3}
2023-11-02T15:25:05.476 DEBUG:teuthology.provision.fog.smithi150:Requesting image smithi_centos_8.stream (ID 3)
2023-11-02T15:25:05.476 DEBUG:teuthology.provision.fog.smithi150:FOG request: PUT /host/255 {"imageID": 3}
2023-11-02T15:25:05.514 INFO:teuthology.provision.fog.smithi150:Scheduling deploy of centos 8.stream
2023-11-02T15:25:05.514 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None
2023-11-02T15:25:05.517 INFO:teuthology.provision.fog.smithi088:Scheduling deploy of centos 8.stream
2023-11-02T15:25:05.517 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-02T15:25:05.681 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /tasktype {"name": "deploy"}
2023-11-02T15:25:05.690 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /tasktype {"name": "deploy"}
2023-11-02T15:25:05.715 DEBUG:teuthology.provision.fog.smithi088:FOG request: POST /host/193/task {"taskTypeID": 1}
2023-11-02T15:25:05.723 DEBUG:teuthology.provision.fog.smithi150:FOG request: POST /host/255/task {"taskTypeID": 1}
2023-11-02T15:25:05.778 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-02T15:25:05.793 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None
2023-11-02T15:25:06.011 INFO:teuthology.orchestra.console.smithi088:Power off
2023-11-02T15:25:06.012 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-02T15:25:06.024 INFO:teuthology.orchestra.console.smithi150:Power off
2023-11-02T15:25:06.024 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-02T15:25:06.033 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off
2023-11-02T15:25:06.034 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-02T15:25:06.045 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off
2023-11-02T15:25:06.046 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-02T15:25:06.055 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on
2023-11-02T15:25:06.066 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on
2023-11-02T15:25:10.058 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-02T15:25:10.067 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-02T15:25:10.167 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on
2023-11-02T15:25:10.175 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on
2023-11-02T15:25:14.170 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-02T15:25:14.177 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-02T15:25:14.279 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off
2023-11-02T15:25:14.285 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on
2023-11-02T15:25:14.381 INFO:teuthology.orchestra.console.smithi088:Power off completed
2023-11-02T15:25:14.482 INFO:teuthology.orchestra.console.smithi088:Power on
2023-11-02T15:25:14.482 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-02T15:25:14.502 DEBUG:teuthology.orchestra.console.smithi088:power on output: Chassis Power Control: Up/On
2023-11-02T15:25:14.503 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-02T15:25:14.523 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off
2023-11-02T15:25:18.286 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-02T15:25:18.395 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off
2023-11-02T15:25:18.495 INFO:teuthology.orchestra.console.smithi150:Power off completed
2023-11-02T15:25:18.524 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-02T15:25:18.596 INFO:teuthology.orchestra.console.smithi150:Power on
2023-11-02T15:25:18.597 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-02T15:25:18.618 DEBUG:teuthology.orchestra.console.smithi150:power on output: Chassis Power Control: Up/On
2023-11-02T15:25:18.618 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-02T15:25:18.633 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off
2023-11-02T15:25:18.640 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off
2023-11-02T15:25:22.638 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-02T15:25:22.645 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-02T15:25:22.747 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on
2023-11-02T15:25:22.753 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on
2023-11-02T15:25:22.849 INFO:teuthology.orchestra.console.smithi088:Power on completed
2023-11-02T15:25:22.854 INFO:teuthology.orchestra.console.smithi150:Power on completed
2023-11-02T15:25:22.950 INFO:teuthology.provision.fog.smithi088:Waiting for deploy to finish
2023-11-02T15:25:22.950 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-02T15:25:22.954 INFO:teuthology.provision.fog.smithi150:Waiting for deploy to finish
2023-11-02T15:25:22.955 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None
2023-11-02T15:25:38.154 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-02T15:25:38.155 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None
2023-11-02T15:25:53.362 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-02T15:25:53.402 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None
2023-11-02T15:26:08.566 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-02T15:26:08.578 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None
2023-11-02T15:26:23.744 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-02T15:26:23.745 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None
2023-11-02T15:26:38.898 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None
2023-11-02T15:26:38.901 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-02T15:26:54.033 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-02T15:26:54.034 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None
2023-11-02T15:27:09.150 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-02T15:27:09.161 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None
2023-11-02T15:27:24.289 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-02T15:27:24.295 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None
2023-11-02T15:27:39.419 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-02T15:27:39.420 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None
2023-11-02T15:27:54.525 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None
2023-11-02T15:27:54.527 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-02T15:28:09.688 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-02T15:28:09.689 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None
2023-11-02T15:28:24.850 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-02T15:28:24.851 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None
2023-11-02T15:28:39.983 DEBUG:teuthology.provision.fog.smithi150:FOG request: GET /task/active None
2023-11-02T15:28:39.986 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-02T15:28:40.124 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:28:43.198 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2023-11-02T15:28:49.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:28:52.282 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2023-11-02T15:28:55.132 DEBUG:teuthology.provision.fog.smithi088:FOG request: GET /task/active None
2023-11-02T15:28:55.245 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:28:58.286 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:28:58.298 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-02T15:29:01.338 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2023-11-02T15:29:04.301 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:29:07.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:29:07.358 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-02T15:29:10.394 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2023-11-02T15:29:13.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:29:16.398 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:29:16.442 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-02T15:29:19.450 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2023-11-02T15:29:22.445 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:29:25.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:29:25.498 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-02T15:29:28.506 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2023-11-02T15:29:31.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:29:34.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:29:34.554 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-02T15:29:37.562 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2023-11-02T15:29:40.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:29:43.567 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:29:43.610 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-02T15:29:46.618 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2023-11-02T15:29:49.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:29:52.620 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:29:52.666 WARNING:teuthology.provision.fog.smithi088:[Errno None] Unable to connect to port 22 on 172.21.15.88
2023-11-02T15:29:58.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:29:58.714 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2023-11-02T15:30:04.721 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:30:07.802 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2023-11-02T15:30:13.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:30:16.858 WARNING:teuthology.provision.fog.smithi150:[Errno None] Unable to connect to port 22 on 172.21.15.150
2023-11-02T15:30:22.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:30:23.577 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-11-02T15:30:23.578 DEBUG:teuthology.orchestra.run.smithi150:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-11-02T15:30:48.325 INFO:teuthology.provision.fog.smithi150:Node is ready
2023-11-02T15:30:48.326 DEBUG:teuthology.orchestra.run.smithi150:> hostname
2023-11-02T15:30:48.350 INFO:teuthology.orchestra.run.smithi150.stdout:smithi150.front.sepia.ceph.com
2023-11-02T15:30:48.351 DEBUG:teuthology.orchestra.run.smithi150:> grep smithi150.front.sepia.ceph.com /etc/hosts
2023-11-02T15:30:48.411 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150 smithi150.front.sepia.ceph.com smithi150
2023-11-02T15:30:48.411 DEBUG:teuthology.orchestra.run.smithi150:> sudo hostname smithi150
2023-11-02T15:30:48.546 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i -e 's/smithi150.front.sepia.ceph.com/smithi150/g' /etc/hosts
2023-11-02T15:30:48.580 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i -e 's/172.21.15.150/172.21.15.150/g' /etc/hosts
2023-11-02T15:30:48.651 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i -e 's/smithi150.front.sepia.ceph.com/smithi150/g' /etc/hostname
2023-11-02T15:30:48.721 DEBUG:teuthology.orchestra.run.smithi150:> sudo hostname smithi150
2023-11-02T15:30:48.790 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release
2023-11-02T15:30:48.848 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream"
2023-11-02T15:30:48.848 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="8"
2023-11-02T15:30:48.848 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos"
2023-11-02T15:30:48.848 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora"
2023-11-02T15:30:48.848 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="8"
2023-11-02T15:30:48.849 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el8"
2023-11-02T15:30:48.849 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 8"
2023-11-02T15:30:48.849 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31"
2023-11-02T15:30:48.849 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-11-02T15:30:48.849 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/"
2023-11-02T15:30:48.850 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-11-02T15:30:48.850 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-11-02T15:30:48.850 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-11-02T15:30:48.851 INFO:teuthology.provision.fog.smithi150:Deploy complete!
2023-11-02T15:30:48.852 DEBUG:teuthology.parallel:result is None
2023-11-02T15:30:58.683 WARNING:teuthology.provision.fog.smithi088:timed out
2023-11-02T15:31:04.686 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:31:04.891 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-11-02T15:31:04.891 DEBUG:teuthology.orchestra.run.smithi088:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-11-02T15:31:05.151 INFO:teuthology.provision.fog.smithi088:Node is ready
2023-11-02T15:31:05.151 DEBUG:teuthology.orchestra.run.smithi088:> hostname
2023-11-02T15:31:05.170 INFO:teuthology.orchestra.run.smithi088.stdout:smithi088.front.sepia.ceph.com
2023-11-02T15:31:05.171 DEBUG:teuthology.orchestra.run.smithi088:> grep smithi088.front.sepia.ceph.com /etc/hosts
2023-11-02T15:31:05.230 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88 smithi088.front.sepia.ceph.com smithi088
2023-11-02T15:31:05.230 DEBUG:teuthology.orchestra.run.smithi088:> sudo hostname smithi088
2023-11-02T15:31:05.345 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i -e 's/smithi088.front.sepia.ceph.com/smithi088/g' /etc/hosts
2023-11-02T15:31:05.375 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i -e 's/172.21.15.88/172.21.15.88/g' /etc/hosts
2023-11-02T15:31:05.445 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i -e 's/smithi088.front.sepia.ceph.com/smithi088/g' /etc/hostname
2023-11-02T15:31:05.513 DEBUG:teuthology.orchestra.run.smithi088:> sudo hostname smithi088
2023-11-02T15:31:05.581 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release
2023-11-02T15:31:05.639 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="CentOS Stream"
2023-11-02T15:31:05.639 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="8"
2023-11-02T15:31:05.639 INFO:teuthology.orchestra.run.smithi088.stdout:ID="centos"
2023-11-02T15:31:05.640 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="rhel fedora"
2023-11-02T15:31:05.640 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="8"
2023-11-02T15:31:05.640 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el8"
2023-11-02T15:31:05.640 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="CentOS Stream 8"
2023-11-02T15:31:05.640 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31"
2023-11-02T15:31:05.640 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-11-02T15:31:05.641 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://centos.org/"
2023-11-02T15:31:05.641 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-11-02T15:31:05.641 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-11-02T15:31:05.641 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-11-02T15:31:05.642 INFO:teuthology.provision.fog.smithi088:Deploy complete!
2023-11-02T15:31:05.642 DEBUG:teuthology.parallel:result is None
2023-11-02T15:31:05.789 INFO:teuthology.lock.ops:Checking smithi088.front.sepia.ceph.com
2023-11-02T15:31:05.790 INFO:teuthology.lock.ops:New key found. Updating...
2023-11-02T15:31:05.815 INFO:teuthology.lock.ops:Checking smithi150.front.sepia.ceph.com
2023-11-02T15:31:05.815 INFO:teuthology.lock.ops:New key found. Updating...
2023-11-02T15:31:05.832 INFO:teuthology.lock.ops:Updating [smithi088.front.sepia.ceph.com]: set os type and version on server
2023-11-02T15:31:05.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com'
2023-11-02T15:31:05.833 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:31:05.898 DEBUG:teuthology.orchestra.run.smithi088:> true
2023-11-02T15:31:05.964 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com'
2023-11-02T15:31:05.964 DEBUG:teuthology.orchestra.run.smithi088:> uname -m
2023-11-02T15:31:06.021 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64
2023-11-02T15:31:06.021 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release
2023-11-02T15:31:06.078 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="CentOS Stream"
2023-11-02T15:31:06.078 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="8"
2023-11-02T15:31:06.078 INFO:teuthology.orchestra.run.smithi088.stdout:ID="centos"
2023-11-02T15:31:06.079 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="rhel fedora"
2023-11-02T15:31:06.079 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="8"
2023-11-02T15:31:06.079 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el8"
2023-11-02T15:31:06.079 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="CentOS Stream 8"
2023-11-02T15:31:06.079 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31"
2023-11-02T15:31:06.080 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-11-02T15:31:06.080 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://centos.org/"
2023-11-02T15:31:06.080 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-11-02T15:31:06.080 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-11-02T15:31:06.080 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-11-02T15:31:06.081 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server
2023-11-02T15:31:06.106 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: set os type and version on server
2023-11-02T15:31:06.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com'
2023-11-02T15:31:06.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-02T15:31:06.179 DEBUG:teuthology.orchestra.run.smithi150:> true
2023-11-02T15:31:06.313 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com'
2023-11-02T15:31:06.313 DEBUG:teuthology.orchestra.run.smithi150:> uname -m
2023-11-02T15:31:06.372 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64
2023-11-02T15:31:06.373 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release
2023-11-02T15:31:06.431 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream"
2023-11-02T15:31:06.431 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="8"
2023-11-02T15:31:06.431 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos"
2023-11-02T15:31:06.432 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora"
2023-11-02T15:31:06.432 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="8"
2023-11-02T15:31:06.432 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el8"
2023-11-02T15:31:06.432 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 8"
2023-11-02T15:31:06.432 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31"
2023-11-02T15:31:06.433 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-11-02T15:31:06.433 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/"
2023-11-02T15:31:06.433 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-11-02T15:31:06.433 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-11-02T15:31:06.433 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-11-02T15:31:06.434 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server
2023-11-02T15:31:06.457 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-02T15:31:06.548 INFO:teuthology.dispatcher.supervisor:Running job 7444772
2023-11-02T15:31:06.548 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2023-11-02_14:25:02-fs:upgrade-main-distro-default-smithi/7444772 --name vshankar-2023-11-02_14:25:02-fs:upgrade-main-distro-default-smithi --description fs:upgrade/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-upgrade 3-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} -- /home/teuthworker/archive/vshankar-2023-11-02_14:25:02-fs:upgrade-main-distro-default-smithi/7444772/orig.config.yaml
2023-11-02T15:31:06.551 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2023-11-02_14:25:02-fs:upgrade-main-distro-default-smithi/7444772
2023-11-02T15:31:06.551 INFO:teuthology.dispatcher.supervisor:Job PID: 408883
2023-11-02T15:31:06.552 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-11-02T15:33:06.651 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-02T15:35:06.827 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-02T15:37:07.005 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-02T15:39:07.190 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-02T15:41:07.427 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-02T15:43:07.607 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-02T15:45:07.779 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-02T15:47:07.951 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-02T15:49:08.119 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-02T15:51:08.283 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-02T15:53:08.418 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-02T15:55:08.579 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-02T15:57:08.747 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-02T15:57:08.821 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120
2023-11-02T15:57:08.941 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-11-02T15:57:08.941 INFO:teuthology.nuke:Checking targets against current locks
2023-11-02T15:57:08.992 DEBUG:teuthology.nuke:shortname: smithi088
2023-11-02T15:57:08.993 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-02T15:57:09.013 DEBUG:teuthology.nuke:shortname: smithi150
2023-11-02T15:57:09.014 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-02T15:57:09.017 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-11-02_14:25:02-fs:upgrade-main-distro-default-smithi/7444772', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-02 15:25:04.388123', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOOt/mb2gkIqbSyuSmurM1PWf1tPP9R0TMpjWoV7uBi3LKY+E/yy5ZF4SzRXlkHqRiFEL1L4Q9NrpdGPQ6lCTtc='}
2023-11-02T15:57:09.041 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-11-02_14:25:02-fs:upgrade-main-distro-default-smithi/7444772', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-02 15:25:04.389912', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBnBtdkjgLDaKaXaF+ipMM+sDuInyZyWmCq2JPg/wXkCV+Ip9w7SQeH7CPKYAfl19V1SK8hSY9YYUzcx/l7qSrw='}
2023-11-02T15:57:09.066 INFO:teuthology.orchestra.console.smithi088:Power off
2023-11-02T15:57:09.067 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-02T15:57:09.088 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off
2023-11-02T15:57:09.089 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-02T15:57:09.105 INFO:teuthology.orchestra.console.smithi150:Power off
2023-11-02T15:57:09.105 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-02T15:57:09.113 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off
2023-11-02T15:57:09.125 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off
2023-11-02T15:57:09.126 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-02T15:57:09.148 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off
2023-11-02T15:57:09.214 INFO:teuthology.orchestra.console.smithi088:Power off completed
2023-11-02T15:57:09.248 INFO:teuthology.orchestra.console.smithi150:Power off completed
2023-11-02T15:57:09.369 INFO:teuthology.lock.ops:unlocked: smithi088.front.sepia.ceph.com
2023-11-02T15:57:09.394 INFO:teuthology.lock.ops:unlocked: smithi150.front.sepia.ceph.com