2024-03-03T05:40:44.912 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55
2024-03-03T05:40:44.913 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-03T05:40:45.066 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f2d2ca21f70>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f2d2da82c10>, signals=[15])
2024-03-03T05:40:45.066 INFO:teuthology.lock.ops:Start node 'smithi090.front.sepia.ceph.com' reimaging
2024-03-03T05:40:45.067 INFO:teuthology.lock.ops:Updating [smithi090.front.sepia.ceph.com]: reset os type and version on server
2024-03-03T05:40:45.067 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server
2024-03-03T05:40:45.084 INFO:teuthology.lock.ops:Start node 'smithi105.front.sepia.ceph.com' reimaging
2024-03-03T05:40:45.085 INFO:teuthology.lock.ops:Updating [smithi105.front.sepia.ceph.com]: reset os type and version on server
2024-03-03T05:40:45.085 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server
2024-03-03T05:40:45.109 INFO:teuthology.lock.ops:Start node 'smithi170.front.sepia.ceph.com' reimaging
2024-03-03T05:40:45.109 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: reset os type and version on server
2024-03-03T05:40:45.110 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server
2024-03-03T05:40:45.200 DEBUG:teuthology.provision.fog.smithi090:Requesting image smithi_rhel_8.6 (ID 5)
2024-03-03T05:40:45.213 DEBUG:teuthology.provision.fog.smithi105:Requesting image smithi_rhel_8.6 (ID 5)
2024-03-03T05:40:45.243 DEBUG:teuthology.provision.fog.smithi170:Requesting image smithi_rhel_8.6 (ID 5)
2024-03-03T05:40:45.246 INFO:teuthology.provision.fog.smithi090:Scheduling deploy of rhel 8.6
2024-03-03T05:40:45.253 INFO:teuthology.provision.fog.smithi105:Scheduling deploy of rhel 8.6
2024-03-03T05:40:45.302 INFO:teuthology.provision.fog.smithi170:Scheduling deploy of rhel 8.6
2024-03-03T05:40:45.511 INFO:teuthology.orchestra.console.smithi105:Power off
2024-03-03T05:40:45.512 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-03T05:40:45.524 INFO:teuthology.orchestra.console.smithi090:Power off
2024-03-03T05:40:45.524 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-03T05:40:45.537 DEBUG:teuthology.orchestra.console.smithi105:power off output: Chassis Power Control: Down/Off
2024-03-03T05:40:45.538 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:40:45.548 DEBUG:teuthology.orchestra.console.smithi090:power off output: Chassis Power Control: Down/Off
2024-03-03T05:40:45.548 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:40:45.565 INFO:teuthology.orchestra.console.smithi170:Power off
2024-03-03T05:40:45.566 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-03T05:40:45.573 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off
2024-03-03T05:40:45.574 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off
2024-03-03T05:40:45.588 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off
2024-03-03T05:40:45.588 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:40:45.612 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on
2024-03-03T05:40:45.674 INFO:teuthology.orchestra.console.smithi090:Power off completed
2024-03-03T05:40:45.675 INFO:teuthology.orchestra.console.smithi105:Power off completed
2024-03-03T05:40:45.775 INFO:teuthology.orchestra.console.smithi090:Power on
2024-03-03T05:40:45.775 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-03T05:40:45.782 INFO:teuthology.orchestra.console.smithi105:Power on
2024-03-03T05:40:45.783 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-03T05:40:45.796 DEBUG:teuthology.orchestra.console.smithi090:power on output: Chassis Power Control: Up/On
2024-03-03T05:40:45.797 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:40:45.805 DEBUG:teuthology.orchestra.console.smithi105:power on output: Chassis Power Control: Up/On
2024-03-03T05:40:45.805 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:40:45.820 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off
2024-03-03T05:40:45.827 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off
2024-03-03T05:40:49.616 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:40:49.725 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on
2024-03-03T05:40:49.821 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:40:49.828 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:40:49.930 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off
2024-03-03T05:40:49.938 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off
2024-03-03T05:40:53.728 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:40:53.837 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off
2024-03-03T05:40:53.930 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:40:53.938 INFO:teuthology.orchestra.console.smithi170:Power off completed
2024-03-03T05:40:53.939 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:40:54.040 INFO:teuthology.orchestra.console.smithi170:Power on
2024-03-03T05:40:54.040 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-03-03T05:40:54.047 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on
2024-03-03T05:40:54.048 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on
2024-03-03T05:40:54.061 DEBUG:teuthology.orchestra.console.smithi170:power on output: Chassis Power Control: Up/On
2024-03-03T05:40:54.062 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:40:54.086 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off
2024-03-03T05:40:54.149 INFO:teuthology.orchestra.console.smithi090:Power on completed
2024-03-03T05:40:54.149 INFO:teuthology.orchestra.console.smithi105:Power on completed
2024-03-03T05:40:54.250 INFO:teuthology.provision.fog.smithi090:Waiting for deploy to finish
2024-03-03T05:40:54.251 INFO:teuthology.provision.fog.smithi105:Waiting for deploy to finish
2024-03-03T05:40:54.340 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:40:54.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:40:57.404 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2024-03-03T05:40:57.404 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2024-03-03T05:40:58.088 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:40:58.197 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off
2024-03-03T05:41:02.200 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:41:02.418 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on
2024-03-03T05:41:02.518 INFO:teuthology.orchestra.console.smithi170:Power on completed
2024-03-03T05:41:02.619 INFO:teuthology.provision.fog.smithi170:Waiting for deploy to finish
2024-03-03T05:41:03.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:41:03.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:41:06.464 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2024-03-03T05:41:06.464 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2024-03-03T05:41:12.468 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:41:12.470 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:41:15.548 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2024-03-03T05:41:15.548 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2024-03-03T05:41:21.552 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:41:21.554 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:41:24.608 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2024-03-03T05:41:24.608 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2024-03-03T05:41:30.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:41:30.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:41:33.696 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2024-03-03T05:41:33.696 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2024-03-03T05:41:39.702 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:41:39.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:41:42.780 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2024-03-03T05:41:42.780 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2024-03-03T05:41:48.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:41:48.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:41:51.836 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2024-03-03T05:41:51.837 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2024-03-03T05:41:57.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:41:57.845 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:42:00.924 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2024-03-03T05:42:00.924 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2024-03-03T05:42:06.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:42:06.929 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:42:09.980 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2024-03-03T05:42:13.024 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2024-03-03T05:42:15.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:42:19.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:42:22.112 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2024-03-03T05:42:28.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:42:28.245 ERROR:teuthology.orchestra.connection:Error authenticating with smithi105.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-03-03T05:42:32.373 ERROR:teuthology.orchestra.connection:Error authenticating with smithi105.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-03-03T05:42:34.300 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2024-03-03T05:42:39.466 ERROR:teuthology.orchestra.connection:Error authenticating with smithi105.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-03-03T05:42:40.301 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:42:43.356 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2024-03-03T05:42:49.358 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:42:49.470 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-03-03T05:42:49.566 ERROR:teuthology.orchestra.connection:Error authenticating with smithi105.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-03-03T05:42:53.627 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-03-03T05:43:00.732 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-03-03T05:43:02.694 ERROR:teuthology.orchestra.connection:Error authenticating with smithi105.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-03-03T05:43:10.829 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-03-03T05:43:18.819 ERROR:teuthology.orchestra.connection:Error authenticating with smithi105.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-03-03T05:43:23.964 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-03-03T05:43:40.099 ERROR:teuthology.orchestra.connection:Error authenticating with smithi090.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-03-03T05:43:48.516 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:43:51.580 WARNING:teuthology.provision.fog.smithi170:[Errno None] Unable to connect to port 22 on 172.21.15.170
2024-03-03T05:43:57.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:44:00.636 WARNING:teuthology.provision.fog.smithi170:[Errno None] Unable to connect to port 22 on 172.21.15.170
2024-03-03T05:44:06.638 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:44:09.692 WARNING:teuthology.provision.fog.smithi170:[Errno None] Unable to connect to port 22 on 172.21.15.170
2024-03-03T05:44:12.188 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2024-03-03T05:44:15.693 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:44:17.471 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2024-03-03T05:44:18.189 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:44:18.752 WARNING:teuthology.provision.fog.smithi170:[Errno None] Unable to connect to port 22 on 172.21.15.170
2024-03-03T05:44:21.244 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2024-03-03T05:44:23.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:44:24.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:44:26.524 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2024-03-03T05:44:27.246 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:44:27.808 WARNING:teuthology.provision.fog.smithi170:[Errno None] Unable to connect to port 22 on 172.21.15.170
2024-03-03T05:44:32.525 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:44:33.809 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:44:35.580 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2024-03-03T05:44:36.860 WARNING:teuthology.provision.fog.smithi170:[Errno None] Unable to connect to port 22 on 172.21.15.170
2024-03-03T05:44:41.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:44:41.724 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2024-03-03T05:44:42.861 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:44:45.628 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2024-03-03T05:44:47.726 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:44:50.780 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2024-03-03T05:44:51.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:44:52.640 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2024-03-03T05:44:56.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:44:58.642 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:44:59.836 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2024-03-03T05:45:01.696 WARNING:teuthology.provision.fog.smithi105:[Errno None] Unable to connect to port 22 on 172.21.15.105
2024-03-03T05:45:05.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:45:05.980 WARNING:teuthology.provision.fog.smithi090:[Errno None] Unable to connect to port 22 on 172.21.15.90
2024-03-03T05:45:07.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:45:07.905 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-03-03T05:45:07.905 DEBUG:teuthology.orchestra.run.smithi105:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-03-03T05:45:11.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-03-03T05:45:28.362 INFO:teuthology.provision.fog.smithi105:Node is ready
2024-03-03T05:45:28.362 DEBUG:teuthology.orchestra.run.smithi105:> hostname
2024-03-03T05:45:28.386 INFO:teuthology.orchestra.run.smithi105.stdout:smithi193
2024-03-03T05:45:28.386 DEBUG:teuthology.orchestra.run.smithi105:> grep smithi193 /etc/hosts
2024-03-03T05:45:28.450 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.193 smithi193.front.sepia.ceph.com smithi193
2024-03-03T05:45:28.450 DEBUG:teuthology.orchestra.run.smithi105:> sudo hostname smithi105
2024-03-03T05:45:28.582 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i -e 's/smithi193/smithi105/g' /etc/hosts
2024-03-03T05:45:28.620 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i -e 's/172.21.15.193/172.21.15.105/g' /etc/hosts
2024-03-03T05:45:28.690 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i -e 's/smithi193/smithi105/g' /etc/hostname
2024-03-03T05:45:28.761 DEBUG:teuthology.orchestra.run.smithi105:> sudo hostname smithi105
2024-03-03T05:45:28.831 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release
2024-03-03T05:45:28.890 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="CentOS Stream"
2024-03-03T05:45:28.890 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="8"
2024-03-03T05:45:28.890 INFO:teuthology.orchestra.run.smithi105.stdout:ID="centos"
2024-03-03T05:45:28.890 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="rhel fedora"
2024-03-03T05:45:28.890 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="8"
2024-03-03T05:45:28.890 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el8"
2024-03-03T05:45:28.890 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="CentOS Stream 8"
2024-03-03T05:45:28.890 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31"
2024-03-03T05:45:28.891 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2024-03-03T05:45:28.891 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://centos.org/"
2024-03-03T05:45:28.891 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-03-03T05:45:28.891 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2024-03-03T05:45:28.891 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-03-03T05:45:28.894 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/provision/fog.py", line 92, in create
    self._verify_installed_os()
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/provision/fog.py", line 347, in _verify_installed_os
    raise RuntimeError(
RuntimeError: Expected smithi105's OS to be rhel 8.6 but found centos 8
2024-03-03T05:45:28.895 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-03-03T05:45:29.222 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=218b1421a1684e75a5df2e722c79f43b
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/provision/fog.py", line 92, in create
    self._verify_installed_os()
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/provision/fog.py", line 347, in _verify_installed_os
    raise RuntimeError(
RuntimeError: Expected smithi105's OS to be rhel 8.6 but found centos 8
2024-03-03T05:45:29.224 INFO:teuthology.nuke:Checking targets against current locks
2024-03-03T05:45:29.293 DEBUG:teuthology.nuke:shortname: smithi090
2024-03-03T05:45:29.293 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-03-03T05:45:29.314 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-02_17:47:35-fs-reef-distro-default-smithi/7579975', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-03 05:40:44.257533', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN2QZxMHSBub6Xm54mZHSdLW1TwyIIGZv4nnOMrV/q2HPngsH5cPB/hYB9oIje4JTyazHuNSgoJ7wH83bzZPJU4='}
2024-03-03T05:45:29.318 DEBUG:teuthology.nuke:shortname: smithi105
2024-03-03T05:45:29.318 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-03-03T05:45:29.325 DEBUG:teuthology.nuke:shortname: smithi170
2024-03-03T05:45:29.325 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-03-03T05:45:29.335 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-02_17:47:35-fs-reef-distro-default-smithi/7579975', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-03 05:40:44.258798', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAk5IaZk3BUZLW8fe2OKJ9wayqgpHvFlPOAyN2hgPT/HGk8dcEhPbXwAlSDU92WfPKRsr9QNmEG+fYJrKV67Xb8='}
2024-03-03T05:45:29.346 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-02_17:47:35-fs-reef-distro-default-smithi/7579974', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-03 05:40:43.848146', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOw614ojOXxC6LVOsEuetuM38oAaTMicjdbG7tALzhf5rknGIc4A0E1FKkZX8Mm9xjLTcrXOr3Hm3akdDvVP5OY='}
2024-03-03T05:45:29.362 INFO:teuthology.orchestra.console.smithi090:Power off
2024-03-03T05:45:29.363 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-03T05:45:29.391 DEBUG:teuthology.orchestra.console.smithi090:power off output: Chassis Power Control: Down/Off
2024-03-03T05:45:29.392 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:45:29.406 INFO:teuthology.orchestra.console.smithi105:Power off
2024-03-03T05:45:29.407 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-03T05:45:29.421 INFO:teuthology.orchestra.console.smithi170:Power off
2024-03-03T05:45:29.421 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-03-03T05:45:29.429 DEBUG:teuthology.orchestra.console.smithi105:power off output: Chassis Power Control: Down/Off
2024-03-03T05:45:29.430 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:45:29.437 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on
2024-03-03T05:45:29.445 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off
2024-03-03T05:45:29.446 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:45:29.454 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on
2024-03-03T05:45:29.469 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on
2024-03-03T05:45:33.438 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:45:33.456 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:45:33.469 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:45:33.552 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on
2024-03-03T05:45:33.565 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on
2024-03-03T05:45:33.579 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on
2024-03-03T05:45:37.553 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:45:37.566 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:45:37.580 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-03-03T05:45:37.663 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off
2024-03-03T05:45:37.675 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off
2024-03-03T05:45:37.690 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off
2024-03-03T05:45:37.765 INFO:teuthology.orchestra.console.smithi090:Power off completed
2024-03-03T05:45:37.776 INFO:teuthology.orchestra.console.smithi105:Power off completed
2024-03-03T05:45:37.792 INFO:teuthology.orchestra.console.smithi170:Power off completed
2024-03-03T05:45:37.923 INFO:teuthology.lock.ops:unlocked: smithi105.front.sepia.ceph.com
2024-03-03T05:45:37.929 INFO:teuthology.lock.ops:unlocked: smithi090.front.sepia.ceph.com
2024-03-03T05:45:37.945 INFO:teuthology.lock.ops:unlocked: smithi170.front.sepia.ceph.com
2024-03-03T05:45:38.314 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/bin/teuthology-dispatcher", line 8, in <module>
    sys.exit(main())
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/scripts/dispatcher.py", line 35, in main
    sys.exit(teuthology.dispatcher.main(args))
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/dispatcher/__init__.py", line 69, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/dispatcher/supervisor.py", line 58, in main
    reimage(job_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/dispatcher/supervisor.py", line 232, in reimage
    reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type'])
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/lock/ops.py", line 325, in reimage_machines
    reimaged[machine] = machines[machine]
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/provision/__init__.py", line 44, in reimage
    result = obj.create()
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/provision/fog.py", line 92, in create
    self._verify_installed_os()
  File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/provision/fog.py", line 347, in _verify_installed_os
    raise RuntimeError(
RuntimeError: Expected smithi105's OS to be rhel 8.6 but found centos 8