2024-04-09T04:35:21.446 INFO:root:teuthology version: 0.0.1.dev267+g6c63784
2024-04-09T04:35:21.447 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-09T04:35:21.632 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fcb21c45f40>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fcb22ca6c10>, signals=[15])
2024-04-09T04:35:21.633 INFO:teuthology.lock.ops:Start node 'smithi076.front.sepia.ceph.com' reimaging
2024-04-09T04:35:21.633 INFO:teuthology.lock.ops:Updating [smithi076.front.sepia.ceph.com]: reset os type and version on server
2024-04-09T04:35:21.633 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server
2024-04-09T04:35:21.652 INFO:teuthology.lock.ops:Start node 'smithi111.front.sepia.ceph.com' reimaging
2024-04-09T04:35:21.652 INFO:teuthology.lock.ops:Updating [smithi111.front.sepia.ceph.com]: reset os type and version on server
2024-04-09T04:35:21.653 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server
2024-04-09T04:35:21.677 INFO:teuthology.lock.ops:Start node 'smithi153.front.sepia.ceph.com' reimaging
2024-04-09T04:35:21.677 INFO:teuthology.lock.ops:Updating [smithi153.front.sepia.ceph.com]: reset os type and version on server
2024-04-09T04:35:21.677 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server
2024-04-09T04:35:21.772 DEBUG:teuthology.provision.fog.smithi076:Requesting image smithi_centos_9.stream (ID 10)
2024-04-09T04:35:21.785 DEBUG:teuthology.provision.fog.smithi111:Requesting image smithi_centos_9.stream (ID 10)
2024-04-09T04:35:21.815 DEBUG:teuthology.provision.fog.smithi153:Requesting image smithi_centos_9.stream (ID 10)
2024-04-09T04:35:21.820 INFO:teuthology.provision.fog.smithi076:Scheduling deploy of centos 9.stream
2024-04-09T04:35:21.828 INFO:teuthology.provision.fog.smithi111:Scheduling deploy of centos 9.stream
2024-04-09T04:35:21.855 INFO:teuthology.provision.fog.smithi153:Scheduling deploy of centos 9.stream
2024-04-09T04:35:22.038 INFO:teuthology.orchestra.console.smithi111:Power off
2024-04-09T04:35:22.038 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-09T04:35:22.061 DEBUG:teuthology.orchestra.console.smithi111:power off output: Chassis Power Control: Down/Off
2024-04-09T04:35:22.062 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-09T04:35:22.083 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on
2024-04-09T04:35:22.089 INFO:teuthology.orchestra.console.smithi076:Power off
2024-04-09T04:35:22.089 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-09T04:35:22.101 INFO:teuthology.orchestra.console.smithi153:Power off
2024-04-09T04:35:22.102 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-04-09T04:35:22.111 DEBUG:teuthology.orchestra.console.smithi076:power off output: Chassis Power Control: Down/Off
2024-04-09T04:35:22.111 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-09T04:35:22.123 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off
2024-04-09T04:35:22.123 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-09T04:35:22.133 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on
2024-04-09T04:35:22.144 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on
2024-04-09T04:35:26.088 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-09T04:35:26.134 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-09T04:35:26.145 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-09T04:35:26.197 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on
2024-04-09T04:35:26.242 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on
2024-04-09T04:35:26.252 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on
2024-04-09T04:35:30.198 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-09T04:35:30.243 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-09T04:35:30.253 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-09T04:35:30.310 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on
2024-04-09T04:35:30.353 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off
2024-04-09T04:35:30.362 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off
2024-04-09T04:35:30.454 INFO:teuthology.orchestra.console.smithi076:Power off completed
2024-04-09T04:35:30.463 INFO:teuthology.orchestra.console.smithi153:Power off completed
2024-04-09T04:35:30.555 INFO:teuthology.orchestra.console.smithi076:Power on
2024-04-09T04:35:30.555 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-04-09T04:35:30.564 INFO:teuthology.orchestra.console.smithi153:Power on
2024-04-09T04:35:30.564 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-04-09T04:35:30.577 DEBUG:teuthology.orchestra.console.smithi076:power on output: Chassis Power Control: Up/On
2024-04-09T04:35:30.578 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-09T04:35:30.584 DEBUG:teuthology.orchestra.console.smithi153:power on output: Chassis Power Control: Up/On
2024-04-09T04:35:30.585 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-09T04:35:30.599 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off
2024-04-09T04:35:30.605 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off
2024-04-09T04:35:34.314 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-09T04:35:34.422 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is off
2024-04-09T04:35:34.523 INFO:teuthology.orchestra.console.smithi111:Power off completed
2024-04-09T04:35:34.600 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-09T04:35:34.607 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-09T04:35:34.624 INFO:teuthology.orchestra.console.smithi111:Power on
2024-04-09T04:35:34.625 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-04-09T04:35:34.644 DEBUG:teuthology.orchestra.console.smithi111:power on output: Chassis Power Control: Up/On
2024-04-09T04:35:34.644 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-09T04:35:34.666 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is off
2024-04-09T04:35:34.708 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off
2024-04-09T04:35:34.714 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off
2024-04-09T04:35:38.670 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-09T04:35:38.710 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-09T04:35:38.717 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-04-09T04:35:38.780 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on
2024-04-09T04:35:38.819 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on
2024-04-09T04:35:38.825 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on
2024-04-09T04:35:38.881 INFO:teuthology.orchestra.console.smithi111:Power on completed
2024-04-09T04:35:38.921 INFO:teuthology.orchestra.console.smithi076:Power on completed
2024-04-09T04:35:38.926 INFO:teuthology.orchestra.console.smithi153:Power on completed
2024-04-09T04:35:38.983 INFO:teuthology.provision.fog.smithi111:Waiting for deploy to finish
2024-04-09T04:35:39.022 INFO:teuthology.provision.fog.smithi076:Waiting for deploy to finish
2024-04-09T04:35:39.027 INFO:teuthology.provision.fog.smithi153:Waiting for deploy to finish
2024-04-09T04:38:10.002 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:38:13.054 WARNING:teuthology.provision.fog.smithi153:[Errno None] Unable to connect to port 22 on 172.21.15.153
2024-04-09T04:38:19.061 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:38:22.142 WARNING:teuthology.provision.fog.smithi153:[Errno None] Unable to connect to port 22 on 172.21.15.153
2024-04-09T04:38:25.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:38:25.168 ERROR:teuthology.orchestra.connection:Error authenticating with smithi076.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2024-04-09T04:38:28.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:38:31.202 WARNING:teuthology.provision.fog.smithi153:[Errno None] Unable to connect to port 22 on 172.21.15.153
2024-04-09T04:38:37.204 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:38:40.216 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:38:40.254 WARNING:teuthology.provision.fog.smithi153:[Errno None] Unable to connect to port 22 on 172.21.15.153
2024-04-09T04:38:43.298 WARNING:teuthology.provision.fog.smithi111:[Errno None] Unable to connect to port 22 on 172.21.15.111
2024-04-09T04:38:46.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:38:49.310 WARNING:teuthology.provision.fog.smithi153:[Errno None] Unable to connect to port 22 on 172.21.15.153
2024-04-09T04:38:49.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:38:52.382 WARNING:teuthology.provision.fog.smithi111:[Errno None] Unable to connect to port 22 on 172.21.15.111
2024-04-09T04:38:55.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:38:58.366 WARNING:teuthology.provision.fog.smithi153:[Errno None] Unable to connect to port 22 on 172.21.15.153
2024-04-09T04:38:58.383 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:39:01.438 WARNING:teuthology.provision.fog.smithi111:[Errno None] Unable to connect to port 22 on 172.21.15.111
2024-04-09T04:39:04.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:39:07.422 WARNING:teuthology.provision.fog.smithi153:[Errno None] Unable to connect to port 22 on 172.21.15.153
2024-04-09T04:39:07.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:39:10.494 WARNING:teuthology.provision.fog.smithi111:[Errno None] Unable to connect to port 22 on 172.21.15.111
2024-04-09T04:39:13.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:39:16.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:39:19.550 WARNING:teuthology.provision.fog.smithi111:[Errno None] Unable to connect to port 22 on 172.21.15.111
2024-04-09T04:39:25.552 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:39:28.606 WARNING:teuthology.provision.fog.smithi111:[Errno None] Unable to connect to port 22 on 172.21.15.111
2024-04-09T04:39:29.172 WARNING:teuthology.provision.fog.smithi076:timed out
2024-04-09T04:39:31.870 WARNING:teuthology.provision.fog.smithi153:[Errno None] Unable to connect to port 22 on 172.21.15.153
2024-04-09T04:39:34.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:39:35.173 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:39:37.662 WARNING:teuthology.provision.fog.smithi111:[Errno None] Unable to connect to port 22 on 172.21.15.111
2024-04-09T04:39:37.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:39:40.926 WARNING:teuthology.provision.fog.smithi153:[Errno None] Unable to connect to port 22 on 172.21.15.153
2024-04-09T04:39:43.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:39:46.718 WARNING:teuthology.provision.fog.smithi111:[Errno None] Unable to connect to port 22 on 172.21.15.111
2024-04-09T04:39:46.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:39:49.982 WARNING:teuthology.provision.fog.smithi153:[Errno None] Unable to connect to port 22 on 172.21.15.153
2024-04-09T04:39:52.719 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:39:55.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:39:56.214 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-04-09T04:39:56.215 DEBUG:teuthology.orchestra.run.smithi153:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-04-09T04:40:16.509 INFO:teuthology.provision.fog.smithi153:Node is ready
2024-04-09T04:40:16.510 DEBUG:teuthology.orchestra.run.smithi153:> hostname
2024-04-09T04:40:16.530 INFO:teuthology.orchestra.run.smithi153.stdout:smithi153.front.sepia.ceph.com
2024-04-09T04:40:16.530 DEBUG:teuthology.orchestra.run.smithi153:> grep smithi153.front.sepia.ceph.com /etc/hosts
2024-04-09T04:40:16.587 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153 smithi153.front.sepia.ceph.com smithi153
2024-04-09T04:40:16.588 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname smithi153
2024-04-09T04:40:16.702 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i -e 's/smithi153.front.sepia.ceph.com/smithi153/g' /etc/hosts
2024-04-09T04:40:16.734 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i -e 's/172.21.15.153/172.21.15.153/g' /etc/hosts
2024-04-09T04:40:16.803 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i -e 's/smithi153.front.sepia.ceph.com/smithi153/g' /etc/hostname
2024-04-09T04:40:16.872 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname smithi153
2024-04-09T04:40:16.939 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release
2024-04-09T04:40:16.994 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream"
2024-04-09T04:40:16.994 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9"
2024-04-09T04:40:16.994 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos"
2024-04-09T04:40:16.994 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora"
2024-04-09T04:40:16.994 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9"
2024-04-09T04:40:16.994 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9"
2024-04-09T04:40:16.995 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9"
2024-04-09T04:40:16.995 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31"
2024-04-09T04:40:16.995 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon"
2024-04-09T04:40:16.995 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-04-09T04:40:16.995 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/"
2024-04-09T04:40:16.995 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-04-09T04:40:16.995 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-04-09T04:40:16.995 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-04-09T04:40:16.996 INFO:teuthology.provision.fog.smithi153:Deploy complete!
2024-04-09T04:40:16.997 DEBUG:teuthology.parallel:result is None
2024-04-09T04:40:35.175 WARNING:teuthology.provision.fog.smithi076:timed out
2024-04-09T04:40:41.177 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:40:41.442 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-04-09T04:40:41.442 DEBUG:teuthology.orchestra.run.smithi076:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-04-09T04:40:46.764 INFO:teuthology.provision.fog.smithi076:Node is ready
2024-04-09T04:40:46.765 DEBUG:teuthology.orchestra.run.smithi076:> hostname
2024-04-09T04:40:46.823 INFO:teuthology.orchestra.run.smithi076.stdout:smithi076.front.sepia.ceph.com
2024-04-09T04:40:46.823 DEBUG:teuthology.orchestra.run.smithi076:> grep smithi076.front.sepia.ceph.com /etc/hosts
2024-04-09T04:40:46.880 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76 smithi076.front.sepia.ceph.com smithi076
2024-04-09T04:40:46.881 DEBUG:teuthology.orchestra.run.smithi076:> sudo hostname smithi076
2024-04-09T04:40:46.984 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i -e 's/smithi076.front.sepia.ceph.com/smithi076/g' /etc/hosts
2024-04-09T04:40:47.016 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i -e 's/172.21.15.76/172.21.15.76/g' /etc/hosts
2024-04-09T04:40:47.087 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i -e 's/smithi076.front.sepia.ceph.com/smithi076/g' /etc/hostname
2024-04-09T04:40:47.158 DEBUG:teuthology.orchestra.run.smithi076:> sudo hostname smithi076
2024-04-09T04:40:47.229 DEBUG:teuthology.orchestra.run.smithi076:> cat /etc/os-release
2024-04-09T04:40:47.285 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="CentOS Stream"
2024-04-09T04:40:47.286 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="9"
2024-04-09T04:40:47.286 INFO:teuthology.orchestra.run.smithi076.stdout:ID="centos"
2024-04-09T04:40:47.286 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="rhel fedora"
2024-04-09T04:40:47.286 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="9"
2024-04-09T04:40:47.286 INFO:teuthology.orchestra.run.smithi076.stdout:PLATFORM_ID="platform:el9"
2024-04-09T04:40:47.286 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="CentOS Stream 9"
2024-04-09T04:40:47.286 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31"
2024-04-09T04:40:47.286 INFO:teuthology.orchestra.run.smithi076.stdout:LOGO="fedora-logo-icon"
2024-04-09T04:40:47.286 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-04-09T04:40:47.286 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://centos.org/"
2024-04-09T04:40:47.286 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-04-09T04:40:47.287 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-04-09T04:40:47.287 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-04-09T04:40:47.287 INFO:teuthology.provision.fog.smithi076:Deploy complete!
2024-04-09T04:40:47.288 DEBUG:teuthology.parallel:result is None
2024-04-09T04:40:52.722 WARNING:teuthology.provision.fog.smithi111:timed out
2024-04-09T04:40:58.723 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:40:58.777 ERROR:paramiko.transport:Unknown exception: The socket has already been closed by another greenlet
2024-04-09T04:40:58.779 ERROR:paramiko.transport:Traceback (most recent call last):
2024-04-09T04:40:58.779 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run
2024-04-09T04:40:58.779 ERROR:paramiko.transport:    ptype, m = self.packetizer.read_message()
2024-04-09T04:40:58.779 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message
2024-04-09T04:40:58.779 ERROR:paramiko.transport:    header = self.read_all(self.__block_size_in, check_rekey=True)
2024-04-09T04:40:58.779 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all
2024-04-09T04:40:58.779 ERROR:paramiko.transport:    x = self.__socket.recv(n)
2024-04-09T04:40:58.779 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 666, in recv
2024-04-09T04:40:58.779 ERROR:paramiko.transport:    self._wait(self._read_event)
2024-04-09T04:40:58.779 ERROR:paramiko.transport:  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
2024-04-09T04:40:58.779 ERROR:paramiko.transport:  File "src/gevent/_hub_primitives.py", line 321, in gevent._gevent_c_hub_primitives.wait_on_socket
2024-04-09T04:40:58.779 ERROR:paramiko.transport:gevent.exceptions.ConcurrentObjectUseError: The socket has already been closed by another greenlet
2024-04-09T04:40:58.779 ERROR:paramiko.transport:
2024-04-09T04:40:59.036 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-04-09T04:40:59.037 DEBUG:teuthology.orchestra.run.smithi111:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-04-09T04:40:59.380 INFO:teuthology.provision.fog.smithi111:Node is ready
2024-04-09T04:40:59.381 DEBUG:teuthology.orchestra.run.smithi111:> hostname
2024-04-09T04:40:59.395 INFO:teuthology.orchestra.run.smithi111.stdout:smithi111.front.sepia.ceph.com
2024-04-09T04:40:59.395 DEBUG:teuthology.orchestra.run.smithi111:> grep smithi111.front.sepia.ceph.com /etc/hosts
2024-04-09T04:40:59.450 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111 smithi111.front.sepia.ceph.com smithi111
2024-04-09T04:40:59.451 DEBUG:teuthology.orchestra.run.smithi111:> sudo hostname smithi111
2024-04-09T04:40:59.551 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i -e 's/smithi111.front.sepia.ceph.com/smithi111/g' /etc/hosts
2024-04-09T04:40:59.579 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i -e 's/172.21.15.111/172.21.15.111/g' /etc/hosts
2024-04-09T04:40:59.646 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i -e 's/smithi111.front.sepia.ceph.com/smithi111/g' /etc/hostname
2024-04-09T04:40:59.714 DEBUG:teuthology.orchestra.run.smithi111:> sudo hostname smithi111
2024-04-09T04:40:59.780 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release
2024-04-09T04:40:59.834 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream"
2024-04-09T04:40:59.834 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="9"
2024-04-09T04:40:59.834 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos"
2024-04-09T04:40:59.834 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora"
2024-04-09T04:40:59.835 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="9"
2024-04-09T04:40:59.835 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el9"
2024-04-09T04:40:59.835 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 9"
2024-04-09T04:40:59.835 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31"
2024-04-09T04:40:59.835 INFO:teuthology.orchestra.run.smithi111.stdout:LOGO="fedora-logo-icon"
2024-04-09T04:40:59.835 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-04-09T04:40:59.835 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/"
2024-04-09T04:40:59.835 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-04-09T04:40:59.835 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-04-09T04:40:59.835 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-04-09T04:40:59.835 INFO:teuthology.provision.fog.smithi111:Deploy complete!
2024-04-09T04:40:59.836 DEBUG:teuthology.parallel:result is None
2024-04-09T04:41:00.045 INFO:teuthology.lock.ops:Checking smithi076.front.sepia.ceph.com
2024-04-09T04:41:00.045 INFO:teuthology.lock.ops:New key found. Updating...
2024-04-09T04:41:00.065 INFO:teuthology.lock.ops:Checking smithi111.front.sepia.ceph.com
2024-04-09T04:41:00.065 INFO:teuthology.lock.ops:New key found. Updating...
2024-04-09T04:41:00.081 INFO:teuthology.lock.ops:Checking smithi153.front.sepia.ceph.com
2024-04-09T04:41:00.082 INFO:teuthology.lock.ops:New key found. Updating...
2024-04-09T04:41:00.100 INFO:teuthology.lock.ops:Updating [smithi076.front.sepia.ceph.com]: set os type and version on server
2024-04-09T04:41:00.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com'
2024-04-09T04:41:00.100 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:41:00.190 DEBUG:teuthology.orchestra.run.smithi076:> true
2024-04-09T04:41:00.360 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com'
2024-04-09T04:41:00.360 DEBUG:teuthology.orchestra.run.smithi076:> uname -m
2024-04-09T04:41:00.414 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64
2024-04-09T04:41:00.414 DEBUG:teuthology.orchestra.run.smithi076:> cat /etc/os-release
2024-04-09T04:41:00.468 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="CentOS Stream"
2024-04-09T04:41:00.468 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="9"
2024-04-09T04:41:00.468 INFO:teuthology.orchestra.run.smithi076.stdout:ID="centos"
2024-04-09T04:41:00.468 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="rhel fedora"
2024-04-09T04:41:00.468 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="9"
2024-04-09T04:41:00.468 INFO:teuthology.orchestra.run.smithi076.stdout:PLATFORM_ID="platform:el9"
2024-04-09T04:41:00.468 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="CentOS Stream 9"
2024-04-09T04:41:00.468 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31"
2024-04-09T04:41:00.468 INFO:teuthology.orchestra.run.smithi076.stdout:LOGO="fedora-logo-icon"
2024-04-09T04:41:00.468 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-04-09T04:41:00.468 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://centos.org/"
2024-04-09T04:41:00.469 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-04-09T04:41:00.469 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-04-09T04:41:00.469 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-04-09T04:41:00.469 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server
2024-04-09T04:41:00.490 INFO:teuthology.lock.ops:Updating [smithi111.front.sepia.ceph.com]: set os type and version on server
2024-04-09T04:41:00.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com'
2024-04-09T04:41:00.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:41:00.581 DEBUG:teuthology.orchestra.run.smithi111:> true
2024-04-09T04:41:00.648 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com'
2024-04-09T04:41:00.648 DEBUG:teuthology.orchestra.run.smithi111:> uname -m
2024-04-09T04:41:00.702 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64
2024-04-09T04:41:00.703 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release
2024-04-09T04:41:00.757 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream"
2024-04-09T04:41:00.757 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="9"
2024-04-09T04:41:00.757 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos"
2024-04-09T04:41:00.757 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora"
2024-04-09T04:41:00.757 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="9"
2024-04-09T04:41:00.757 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el9"
2024-04-09T04:41:00.757 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 9"
2024-04-09T04:41:00.757 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31"
2024-04-09T04:41:00.758 INFO:teuthology.orchestra.run.smithi111.stdout:LOGO="fedora-logo-icon"
2024-04-09T04:41:00.758 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-04-09T04:41:00.758 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/"
2024-04-09T04:41:00.758 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-04-09T04:41:00.758 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-04-09T04:41:00.758 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-04-09T04:41:00.758 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server
2024-04-09T04:41:00.783 INFO:teuthology.lock.ops:Updating [smithi153.front.sepia.ceph.com]: set os type and version on server
2024-04-09T04:41:00.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com'
2024-04-09T04:41:00.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-04-09T04:41:00.870 DEBUG:teuthology.orchestra.run.smithi153:> true
2024-04-09T04:41:00.943 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com'
2024-04-09T04:41:00.943 DEBUG:teuthology.orchestra.run.smithi153:> uname -m
2024-04-09T04:41:00.999 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64
2024-04-09T04:41:01.000 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release
2024-04-09T04:41:01.055 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream"
2024-04-09T04:41:01.055 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9"
2024-04-09T04:41:01.055 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos"
2024-04-09T04:41:01.055 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora"
2024-04-09T04:41:01.055 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9"
2024-04-09T04:41:01.055 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9"
2024-04-09T04:41:01.055 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9"
2024-04-09T04:41:01.056 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31"
2024-04-09T04:41:01.056 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon"
2024-04-09T04:41:01.056 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-04-09T04:41:01.056 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/"
2024-04-09T04:41:01.056 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-04-09T04:41:01.056 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-04-09T04:41:01.056 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-04-09T04:41:01.056 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server
2024-04-09T04:41:01.082 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-09T04:41:01.168 INFO:teuthology.dispatcher.supervisor:Running job 7648013
2024-04-09T04:41:01.168 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/bin/teuthology -v --owner scheduled_adking@teuthology --archive /home/teuthworker/archive/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648013 --name adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi --description orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} -- /home/teuthworker/archive/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648013/orig.config.yaml
2024-04-09T04:41:01.171 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/adking-2024-04-09_04:09:23-orch:cephadm-wip-adk-testing-2024-04-08-1615-distro-default-smithi/7648013
2024-04-09T04:41:01.172 INFO:teuthology.dispatcher.supervisor:Job PID: 174922
2024-04-09T04:41:01.172 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2024-04-09T04:43:01.174 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-09T04:45:01.253 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-09T04:47:01.321 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-09T04:49:01.397 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-09T04:51:01.462 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-09T04:53:01.533 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-09T04:55:01.612 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-09T04:57:01.687 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-09T04:59:01.765 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-09T05:01:01.835 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-09T05:03:01.914 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-04-09T05:03:01.990 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1