2024-01-25T02:30:35.723 INFO:root:teuthology version: 0.0.1.dev251+gc856e0f
2024-01-25T02:30:35.723 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T02:30:35.862 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f02bac6ef40>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f02bbcd4c10>, signals=[15])
2024-01-25T02:30:35.862 INFO:teuthology.lock.ops:Start node 'smithi037.front.sepia.ceph.com' reimaging
2024-01-25T02:30:35.863 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: reset os type and version on server
2024-01-25T02:30:35.863 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server
2024-01-25T02:30:35.886 INFO:teuthology.lock.ops:Start node 'smithi143.front.sepia.ceph.com' reimaging
2024-01-25T02:30:35.886 INFO:teuthology.lock.ops:Updating [smithi143.front.sepia.ceph.com]: reset os type and version on server
2024-01-25T02:30:35.886 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server
2024-01-25T02:30:36.014 DEBUG:teuthology.provision.fog.smithi037:Requesting image smithi_centos_9.stream (ID 10)
2024-01-25T02:30:36.029 DEBUG:teuthology.provision.fog.smithi143:Requesting image smithi_centos_9.stream (ID 10)
2024-01-25T02:30:36.069 INFO:teuthology.provision.fog.smithi143:Scheduling deploy of centos 9.stream
2024-01-25T02:30:36.070 INFO:teuthology.provision.fog.smithi037:Scheduling deploy of centos 9.stream
2024-01-25T02:30:36.289 INFO:teuthology.orchestra.console.smithi143:Power off
2024-01-25T02:30:36.289 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-25T02:30:36.302 INFO:teuthology.orchestra.console.smithi037:Power off
2024-01-25T02:30:36.302 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-25T02:30:36.312 DEBUG:teuthology.orchestra.console.smithi143:power off output: Chassis Power Control: Down/Off
2024-01-25T02:30:36.312 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-25T02:30:36.326 DEBUG:teuthology.orchestra.console.smithi037:power off output: Chassis Power Control: Down/Off
2024-01-25T02:30:36.326 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-25T02:30:36.333 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on
2024-01-25T02:30:36.349 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on
2024-01-25T02:30:40.336 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-25T02:30:40.349 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-25T02:30:40.445 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on
2024-01-25T02:30:40.458 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on
2024-01-25T02:30:44.451 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-25T02:30:44.459 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-25T02:30:44.561 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is off
2024-01-25T02:30:44.568 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off
2024-01-25T02:30:44.662 INFO:teuthology.orchestra.console.smithi143:Power off completed
2024-01-25T02:30:44.669 INFO:teuthology.orchestra.console.smithi037:Power off completed
2024-01-25T02:30:44.763 INFO:teuthology.orchestra.console.smithi143:Power on
2024-01-25T02:30:44.764 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-01-25T02:30:44.771 INFO:teuthology.orchestra.console.smithi037:Power on
2024-01-25T02:30:44.771 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-01-25T02:30:44.784 DEBUG:teuthology.orchestra.console.smithi143:power on output: Chassis Power Control: Up/On
2024-01-25T02:30:44.785 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-25T02:30:44.793 DEBUG:teuthology.orchestra.console.smithi037:power on output: Chassis Power Control: Up/On
2024-01-25T02:30:44.793 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-25T02:30:44.805 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is off
2024-01-25T02:30:44.813 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off
2024-01-25T02:30:48.808 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-25T02:30:48.815 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-25T02:30:48.917 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is off
2024-01-25T02:30:48.923 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off
2024-01-25T02:30:52.922 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-25T02:30:52.931 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-25T02:30:53.033 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on
2024-01-25T02:30:53.039 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on
2024-01-25T02:30:53.134 INFO:teuthology.orchestra.console.smithi143:Power on completed
2024-01-25T02:30:53.140 INFO:teuthology.orchestra.console.smithi037:Power on completed
2024-01-25T02:30:53.235 INFO:teuthology.provision.fog.smithi143:Waiting for deploy to finish
2024-01-25T02:30:53.241 INFO:teuthology.provision.fog.smithi037:Waiting for deploy to finish
2024-01-25T02:33:08.962 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-25T02:33:24.052 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-25T02:33:27.356 WARNING:teuthology.provision.fog.smithi037:[Errno None] Unable to connect to port 22 on 172.21.15.37
2024-01-25T02:33:33.362 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-25T02:33:36.444 WARNING:teuthology.provision.fog.smithi037:[Errno None] Unable to connect to port 22 on 172.21.15.37
2024-01-25T02:33:42.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-25T02:33:42.464 WARNING:teuthology.provision.fog.smithi143:[Errno None] Unable to connect to port 22 on 172.21.15.143
2024-01-25T02:33:45.536 WARNING:teuthology.provision.fog.smithi037:[Errno None] Unable to connect to port 22 on 172.21.15.37
2024-01-25T02:33:48.468 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-25T02:33:51.539 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-25T02:33:51.548 WARNING:teuthology.provision.fog.smithi143:[Errno None] Unable to connect to port 22 on 172.21.15.143
2024-01-25T02:33:57.552 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-25T02:34:00.608 WARNING:teuthology.provision.fog.smithi143:[Errno None] Unable to connect to port 22 on 172.21.15.143
2024-01-25T02:34:06.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-25T02:34:09.692 WARNING:teuthology.provision.fog.smithi143:[Errno None] Unable to connect to port 22 on 172.21.15.143
2024-01-25T02:34:15.696 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-25T02:34:18.748 WARNING:teuthology.provision.fog.smithi143:[Errno None] Unable to connect to port 22 on 172.21.15.143
2024-01-25T02:34:24.755 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-25T02:34:43.132 WARNING:teuthology.provision.fog.smithi143:[Errno None] Unable to connect to port 22 on 172.21.15.143
2024-01-25T02:34:49.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-25T02:34:51.544 WARNING:teuthology.provision.fog.smithi037:timed out
2024-01-25T02:34:52.188 WARNING:teuthology.provision.fog.smithi143:[Errno None] Unable to connect to port 22 on 172.21.15.143
2024-01-25T02:34:57.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-25T02:34:57.857 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-01-25T02:34:57.858 DEBUG:teuthology.orchestra.run.smithi037:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-01-25T02:34:58.189 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-25T02:34:58.195 INFO:teuthology.provision.fog.smithi037:Node is ready
2024-01-25T02:34:58.196 DEBUG:teuthology.orchestra.run.smithi037:> hostname
2024-01-25T02:34:58.210 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037.front.sepia.ceph.com
2024-01-25T02:34:58.210 DEBUG:teuthology.orchestra.run.smithi037:> grep smithi037.front.sepia.ceph.com /etc/hosts
2024-01-25T02:34:58.266 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37 smithi037.front.sepia.ceph.com smithi037
2024-01-25T02:34:58.266 DEBUG:teuthology.orchestra.run.smithi037:> sudo hostname smithi037
2024-01-25T02:34:58.367 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i -e 's/smithi037.front.sepia.ceph.com/smithi037/g' /etc/hosts
2024-01-25T02:34:58.399 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i -e 's/172.21.15.37/172.21.15.37/g' /etc/hosts
2024-01-25T02:34:58.469 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i -e 's/smithi037.front.sepia.ceph.com/smithi037/g' /etc/hostname
2024-01-25T02:34:58.540 DEBUG:teuthology.orchestra.run.smithi037:> sudo hostname smithi037
2024-01-25T02:34:58.610 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release
2024-01-25T02:34:58.665 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="CentOS Stream"
2024-01-25T02:34:58.665 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="9"
2024-01-25T02:34:58.665 INFO:teuthology.orchestra.run.smithi037.stdout:ID="centos"
2024-01-25T02:34:58.666 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="rhel fedora"
2024-01-25T02:34:58.666 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="9"
2024-01-25T02:34:58.666 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el9"
2024-01-25T02:34:58.666 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="CentOS Stream 9"
2024-01-25T02:34:58.666 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31"
2024-01-25T02:34:58.666 INFO:teuthology.orchestra.run.smithi037.stdout:LOGO="fedora-logo-icon"
2024-01-25T02:34:58.666 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-01-25T02:34:58.666 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://centos.org/"
2024-01-25T02:34:58.666 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-01-25T02:34:58.666 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-01-25T02:34:58.666 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-01-25T02:34:58.667 INFO:teuthology.provision.fog.smithi037:Deploy complete!
2024-01-25T02:34:58.669 DEBUG:teuthology.parallel:result is None
2024-01-25T02:35:01.244 WARNING:teuthology.provision.fog.smithi143:[Errno None] Unable to connect to port 22 on 172.21.15.143
2024-01-25T02:35:07.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-25T02:35:07.673 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-01-25T02:35:07.674 DEBUG:teuthology.orchestra.run.smithi143:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-01-25T02:35:23.055 INFO:teuthology.provision.fog.smithi143:Node is ready
2024-01-25T02:35:23.055 DEBUG:teuthology.orchestra.run.smithi143:> hostname
2024-01-25T02:35:23.075 INFO:teuthology.orchestra.run.smithi143.stdout:smithi046
2024-01-25T02:35:23.076 DEBUG:teuthology.orchestra.run.smithi143:> grep smithi046 /etc/hosts
2024-01-25T02:35:23.130 INFO:teuthology.orchestra.run.smithi143.stdout:172.21.15.46 smithi046.front.sepia.ceph.com smithi046
2024-01-25T02:35:23.130 DEBUG:teuthology.orchestra.run.smithi143:> sudo hostname smithi143
2024-01-25T02:35:23.279 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i -e 's/smithi046/smithi143/g' /etc/hosts
2024-01-25T02:35:23.309 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i -e 's/172.21.15.46/172.21.15.143/g' /etc/hosts
2024-01-25T02:35:23.379 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i -e 's/smithi046/smithi143/g' /etc/hostname
2024-01-25T02:35:23.448 DEBUG:teuthology.orchestra.run.smithi143:> sudo hostname smithi143
2024-01-25T02:35:23.516 DEBUG:teuthology.orchestra.run.smithi143:> cat /etc/os-release
2024-01-25T02:35:23.573 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="CentOS Stream"
2024-01-25T02:35:23.573 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="9"
2024-01-25T02:35:23.573 INFO:teuthology.orchestra.run.smithi143.stdout:ID="centos"
2024-01-25T02:35:23.573 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE="rhel fedora"
2024-01-25T02:35:23.573 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="9"
2024-01-25T02:35:23.573 INFO:teuthology.orchestra.run.smithi143.stdout:PLATFORM_ID="platform:el9"
2024-01-25T02:35:23.573 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="CentOS Stream 9"
2024-01-25T02:35:23.574 INFO:teuthology.orchestra.run.smithi143.stdout:ANSI_COLOR="0;31"
2024-01-25T02:35:23.574 INFO:teuthology.orchestra.run.smithi143.stdout:LOGO="fedora-logo-icon"
2024-01-25T02:35:23.574 INFO:teuthology.orchestra.run.smithi143.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-01-25T02:35:23.574 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://centos.org/"
2024-01-25T02:35:23.574 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-01-25T02:35:23.574 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-01-25T02:35:23.574 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-01-25T02:35:23.574 INFO:teuthology.provision.fog.smithi143:Deploy complete!
2024-01-25T02:35:23.575 DEBUG:teuthology.parallel:result is None
2024-01-25T02:35:23.749 INFO:teuthology.lock.ops:Checking smithi037.front.sepia.ceph.com
2024-01-25T02:35:23.749 INFO:teuthology.lock.ops:New key found. Updating...
2024-01-25T02:35:23.778 INFO:teuthology.lock.ops:Checking smithi143.front.sepia.ceph.com
2024-01-25T02:35:23.778 INFO:teuthology.lock.ops:New key found. Updating...
2024-01-25T02:35:23.795 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: set os type and version on server
2024-01-25T02:35:23.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2024-01-25T02:35:23.797 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-25T02:35:23.903 DEBUG:teuthology.orchestra.run.smithi037:> true
2024-01-25T02:35:24.083 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com'
2024-01-25T02:35:24.083 DEBUG:teuthology.orchestra.run.smithi037:> uname -m
2024-01-25T02:35:24.137 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64
2024-01-25T02:35:24.138 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release
2024-01-25T02:35:24.193 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="CentOS Stream"
2024-01-25T02:35:24.193 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="9"
2024-01-25T02:35:24.193 INFO:teuthology.orchestra.run.smithi037.stdout:ID="centos"
2024-01-25T02:35:24.193 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="rhel fedora"
2024-01-25T02:35:24.193 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="9"
2024-01-25T02:35:24.193 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el9"
2024-01-25T02:35:24.193 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="CentOS Stream 9"
2024-01-25T02:35:24.193 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31"
2024-01-25T02:35:24.193 INFO:teuthology.orchestra.run.smithi037.stdout:LOGO="fedora-logo-icon"
2024-01-25T02:35:24.193 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-01-25T02:35:24.193 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://centos.org/"
2024-01-25T02:35:24.194 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-01-25T02:35:24.194 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-01-25T02:35:24.194 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-01-25T02:35:24.194 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server
2024-01-25T02:35:24.219 INFO:teuthology.lock.ops:Updating [smithi143.front.sepia.ceph.com]: set os type and version on server
2024-01-25T02:35:24.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com'
2024-01-25T02:35:24.220 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-01-25T02:35:24.310 DEBUG:teuthology.orchestra.run.smithi143:> true
2024-01-25T02:35:24.374 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com'
2024-01-25T02:35:24.374 DEBUG:teuthology.orchestra.run.smithi143:> uname -m
2024-01-25T02:35:24.429 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64
2024-01-25T02:35:24.429 DEBUG:teuthology.orchestra.run.smithi143:> cat /etc/os-release
2024-01-25T02:35:24.484 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="CentOS Stream"
2024-01-25T02:35:24.484 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="9"
2024-01-25T02:35:24.485 INFO:teuthology.orchestra.run.smithi143.stdout:ID="centos"
2024-01-25T02:35:24.485 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE="rhel fedora"
2024-01-25T02:35:24.485 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="9"
2024-01-25T02:35:24.485 INFO:teuthology.orchestra.run.smithi143.stdout:PLATFORM_ID="platform:el9"
2024-01-25T02:35:24.485 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="CentOS Stream 9"
2024-01-25T02:35:24.485 INFO:teuthology.orchestra.run.smithi143.stdout:ANSI_COLOR="0;31"
2024-01-25T02:35:24.485 INFO:teuthology.orchestra.run.smithi143.stdout:LOGO="fedora-logo-icon"
2024-01-25T02:35:24.486 INFO:teuthology.orchestra.run.smithi143.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2024-01-25T02:35:24.486 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://centos.org/"
2024-01-25T02:35:24.486 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2024-01-25T02:35:24.486 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2024-01-25T02:35:24.486 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2024-01-25T02:35:24.487 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server
2024-01-25T02:35:24.511 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T02:35:24.601 INFO:teuthology.dispatcher.supervisor:Running job 7531150
2024-01-25T02:35:24.601 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/virtualenv/bin/teuthology -v --owner scheduled_cbodley@teuthology --archive /home/teuthworker/archive/cbodley-2024-01-25_02:21:59-rgw-wip-rgw-surface-neorados-distro-default-smithi/7531150 --name cbodley-2024-01-25_02:21:59-rgw-wip-rgw-surface-neorados-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_latest} tasks/test_multi} -- /home/teuthworker/archive/cbodley-2024-01-25_02:21:59-rgw-wip-rgw-surface-neorados-distro-default-smithi/7531150/orig.config.yaml
2024-01-25T02:35:24.603 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/cbodley-2024-01-25_02:21:59-rgw-wip-rgw-surface-neorados-distro-default-smithi/7531150
2024-01-25T02:35:24.604 INFO:teuthology.dispatcher.supervisor:Job PID: 2707738
2024-01-25T02:35:24.604 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2024-01-25T02:37:24.705 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T02:39:24.877 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T02:41:25.045 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T02:43:25.217 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T02:45:25.393 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T02:47:25.569 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T02:49:25.732 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T02:51:25.905 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T02:53:26.080 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T02:55:26.270 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T02:57:26.445 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T02:59:26.616 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:01:26.785 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:03:26.958 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:05:27.121 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:07:27.281 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:09:27.445 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:11:27.619 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:13:27.793 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:15:27.975 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:17:28.143 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:19:28.405 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:21:28.572 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:23:28.733 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:25:28.888 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:27:29.058 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:29:29.216 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:31:29.384 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:33:29.552 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:35:29.728 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:37:29.896 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:39:30.065 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:41:30.234 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:43:30.400 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:45:30.556 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:47:30.684 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:49:30.861 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:51:31.028 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:53:31.163 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:55:31.324 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:57:31.492 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T03:59:31.668 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:01:31.841 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:03:32.017 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:05:32.274 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:07:32.415 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:09:32.589 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:11:32.746 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:13:32.917 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:15:33.093 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:17:33.257 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:19:33.421 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:21:33.596 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:23:33.763 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:25:33.925 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:27:34.076 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:29:34.245 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:31:34.406 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:33:34.587 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:35:34.748 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:37:34.927 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:39:35.081 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:41:35.245 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:43:35.508 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:45:35.671 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:47:35.837 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:49:36.005 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:51:36.177 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:53:36.345 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:55:36.517 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:57:36.685 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T04:59:36.838 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T05:01:36.990 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T05:03:37.164 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T05:05:37.346 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T05:07:37.497 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T05:09:37.666 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T05:11:37.830 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T05:13:38.011 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T05:15:38.177 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T05:17:38.349 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T05:19:38.517 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T05:21:38.687 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T05:23:38.865 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-01-25T05:23:38.933 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1
2024-01-25T05:23:39.103 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2024-01-25T05:23:39.103 INFO:teuthology.nuke:Checking targets against current locks
2024-01-25T05:23:39.152 DEBUG:teuthology.nuke:shortname: smithi037
2024-01-25T05:23:39.152 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-01-25T05:23:39.170 DEBUG:teuthology.nuke:shortname: smithi143
2024-01-25T05:23:39.170 INFO:teuthology.task.internal.check_lock:Checking locks...
2024-01-25T05:23:39.173 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/cbodley-2024-01-25_02:21:59-rgw-wip-rgw-surface-neorados-distro-default-smithi/7531150', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-25 02:30:34.681762', 'locked_by': 'scheduled_cbodley@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP3O+Njyl/g08iU48yTQcxP+Zd0li6Wc24+t37QKQF2ZkeGYi44SW+iWDjsj6BVAIWLjFzgkxsn69hlvNA0WYTU='}
2024-01-25T05:23:39.193 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/cbodley-2024-01-25_02:21:59-rgw-wip-rgw-surface-neorados-distro-default-smithi/7531150', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-25 02:30:34.680461', 'locked_by': 'scheduled_cbodley@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFMWjren5Ha0ApEh2tU44YGpUFmRaPfcI3YG0n3JSrnXv5PcwneqE9CQWOObGTfZv/i8NzfmuhvZb5apWBoi4dE='}
2024-01-25T05:23:39.220 INFO:teuthology.orchestra.console.smithi037:Power off
2024-01-25T05:23:39.221 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-25T05:23:39.243 DEBUG:teuthology.orchestra.console.smithi037:power off output: Chassis Power Control: Down/Off
2024-01-25T05:23:39.244 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-25T05:23:39.256 INFO:teuthology.orchestra.console.smithi143:Power off
2024-01-25T05:23:39.256 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-01-25T05:23:39.264 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off
2024-01-25T05:23:39.279 DEBUG:teuthology.orchestra.console.smithi143:power off output: Chassis Power Control: Down/Off
2024-01-25T05:23:39.280 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-01-25T05:23:39.299 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is off
2024-01-25T05:23:39.367 INFO:teuthology.orchestra.console.smithi037:Power off completed
2024-01-25T05:23:39.400 INFO:teuthology.orchestra.console.smithi143:Power off completed
2024-01-25T05:23:39.525 INFO:teuthology.lock.ops:unlocked: smithi037.front.sepia.ceph.com
2024-01-25T05:23:39.540 INFO:teuthology.lock.ops:unlocked: smithi143.front.sepia.ceph.com