2024-04-08T19:39:21.792 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-08T19:39:21.792 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T19:39:21.943 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fab4fa68c10>, signals=[15]) 2024-04-08T19:39:21.943 INFO:teuthology.lock.ops:Start node 'smithi062.front.sepia.ceph.com' reimaging 2024-04-08T19:39:21.943 INFO:teuthology.lock.ops:Updating [smithi062.front.sepia.ceph.com]: reset os type and version on server 2024-04-08T19:39:21.943 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2024-04-08T19:39:21.961 INFO:teuthology.lock.ops:Start node 'smithi087.front.sepia.ceph.com' reimaging 2024-04-08T19:39:21.962 INFO:teuthology.lock.ops:Updating [smithi087.front.sepia.ceph.com]: reset os type and version on server 2024-04-08T19:39:21.962 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2024-04-08T19:39:22.082 DEBUG:teuthology.provision.fog.smithi062:Requesting image smithi_centos_9.stream (ID 10) 2024-04-08T19:39:22.099 DEBUG:teuthology.provision.fog.smithi087:Requesting image smithi_centos_9.stream (ID 10) 2024-04-08T19:39:22.136 INFO:teuthology.provision.fog.smithi062:Scheduling deploy of centos 9.stream 2024-04-08T19:39:22.139 INFO:teuthology.provision.fog.smithi087:Scheduling deploy of centos 9.stream 2024-04-08T19:39:22.364 INFO:teuthology.orchestra.console.smithi062:Power off 2024-04-08T19:39:22.365 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-08T19:39:22.380 INFO:teuthology.orchestra.console.smithi087:Power off 2024-04-08T19:39:22.381 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-08T19:39:22.391 DEBUG:teuthology.orchestra.console.smithi062:power off output: Chassis Power Control: Down/Off 2024-04-08T19:39:22.391 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T19:39:22.406 DEBUG:teuthology.orchestra.console.smithi087:power off output: Chassis Power Control: Down/Off 2024-04-08T19:39:22.407 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T19:39:22.418 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-04-08T19:39:22.430 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2024-04-08T19:39:26.422 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T19:39:26.431 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T19:39:26.531 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-04-08T19:39:26.539 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2024-04-08T19:39:30.534 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T19:39:30.542 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T19:39:30.645 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-04-08T19:39:30.652 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2024-04-08T19:39:34.650 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T19:39:34.657 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T19:39:34.760 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is off 2024-04-08T19:39:34.766 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is off 2024-04-08T19:39:34.861 INFO:teuthology.orchestra.console.smithi062:Power off completed 2024-04-08T19:39:34.866 INFO:teuthology.orchestra.console.smithi087:Power off completed 2024-04-08T19:39:34.962 INFO:teuthology.orchestra.console.smithi062:Power on 2024-04-08T19:39:34.962 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-08T19:39:34.971 INFO:teuthology.orchestra.console.smithi087:Power on 2024-04-08T19:39:34.972 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-08T19:39:34.986 DEBUG:teuthology.orchestra.console.smithi062:power on output: Chassis Power Control: Up/On 2024-04-08T19:39:34.987 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T19:39:34.995 DEBUG:teuthology.orchestra.console.smithi087:power on output: Chassis Power Control: Up/On 2024-04-08T19:39:34.995 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T19:39:35.010 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is off 2024-04-08T19:39:35.017 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is off 2024-04-08T19:39:39.014 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T19:39:39.022 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T19:39:39.124 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-04-08T19:39:39.130 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is off 2024-04-08T19:39:39.225 INFO:teuthology.orchestra.console.smithi062:Power on completed 2024-04-08T19:39:39.326 INFO:teuthology.provision.fog.smithi062:Waiting for deploy to finish 2024-04-08T19:39:43.134 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-08T19:39:43.244 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2024-04-08T19:39:43.344 INFO:teuthology.orchestra.console.smithi087:Power on completed 2024-04-08T19:39:43.445 INFO:teuthology.provision.fog.smithi087:Waiting for deploy to finish 2024-04-08T19:42:14.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-08T19:42:25.305 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-08T19:42:32.734 WARNING:teuthology.provision.fog.smithi087:[Errno None] Unable to connect to port 22 on 172.21.15.87 2024-04-08T19:42:38.739 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-08T19:42:41.790 WARNING:teuthology.provision.fog.smithi087:[Errno None] Unable to connect to port 22 on 172.21.15.87 2024-04-08T19:42:43.742 WARNING:teuthology.provision.fog.smithi062:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-04-08T19:42:47.794 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-08T19:42:49.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-08T19:42:50.878 WARNING:teuthology.provision.fog.smithi087:[Errno None] Unable to connect to port 22 on 172.21.15.87 2024-04-08T19:42:52.798 WARNING:teuthology.provision.fog.smithi062:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-04-08T19:42:56.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-08T19:42:58.802 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-08T19:42:59.934 WARNING:teuthology.provision.fog.smithi087:[Errno None] Unable to connect to port 22 on 172.21.15.87 2024-04-08T19:43:01.854 WARNING:teuthology.provision.fog.smithi062:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-04-08T19:43:05.938 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-08T19:43:07.858 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-08T19:43:08.990 WARNING:teuthology.provision.fog.smithi087:[Errno None] Unable to connect to port 22 on 172.21.15.87 2024-04-08T19:43:10.914 WARNING:teuthology.provision.fog.smithi062:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-04-08T19:43:14.996 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-08T19:43:16.917 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-08T19:44:15.055 WARNING:teuthology.provision.fog.smithi087:timed out 2024-04-08T19:44:16.921 WARNING:teuthology.provision.fog.smithi062:timed out 2024-04-08T19:44:21.059 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-08T19:44:21.297 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-04-08T19:44:21.297 DEBUG:teuthology.orchestra.run.smithi087:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-04-08T19:44:21.599 INFO:teuthology.provision.fog.smithi087:Node is ready 2024-04-08T19:44:21.600 DEBUG:teuthology.orchestra.run.smithi087:> hostname 2024-04-08T19:44:21.613 INFO:teuthology.orchestra.run.smithi087.stdout:smithi087.front.sepia.ceph.com 2024-04-08T19:44:21.614 DEBUG:teuthology.orchestra.run.smithi087:> grep smithi087.front.sepia.ceph.com /etc/hosts 2024-04-08T19:44:21.668 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87 smithi087.front.sepia.ceph.com smithi087 2024-04-08T19:44:21.668 DEBUG:teuthology.orchestra.run.smithi087:> sudo hostname smithi087 2024-04-08T19:44:21.786 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i -e 's/smithi087.front.sepia.ceph.com/smithi087/g' /etc/hosts 2024-04-08T19:44:21.814 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i -e 's/172.21.15.87/172.21.15.87/g' /etc/hosts 2024-04-08T19:44:21.881 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i -e 's/smithi087.front.sepia.ceph.com/smithi087/g' /etc/hostname 2024-04-08T19:44:21.948 DEBUG:teuthology.orchestra.run.smithi087:> sudo hostname smithi087 2024-04-08T19:44:22.015 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2024-04-08T19:44:22.069 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="CentOS Stream" 2024-04-08T19:44:22.069 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="9" 2024-04-08T19:44:22.070 INFO:teuthology.orchestra.run.smithi087.stdout:ID="centos" 2024-04-08T19:44:22.070 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE="rhel fedora" 2024-04-08T19:44:22.070 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="9" 2024-04-08T19:44:22.070 INFO:teuthology.orchestra.run.smithi087.stdout:PLATFORM_ID="platform:el9" 2024-04-08T19:44:22.070 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-08T19:44:22.070 INFO:teuthology.orchestra.run.smithi087.stdout:ANSI_COLOR="0;31" 2024-04-08T19:44:22.070 INFO:teuthology.orchestra.run.smithi087.stdout:LOGO="fedora-logo-icon" 2024-04-08T19:44:22.070 INFO:teuthology.orchestra.run.smithi087.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-08T19:44:22.070 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://centos.org/" 2024-04-08T19:44:22.070 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-08T19:44:22.070 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-08T19:44:22.070 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-08T19:44:22.071 INFO:teuthology.provision.fog.smithi087:Deploy complete! 2024-04-08T19:44:22.072 DEBUG:teuthology.parallel:result is None 2024-04-08T19:44:22.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-08T19:44:23.173 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-04-08T19:44:23.173 DEBUG:teuthology.orchestra.run.smithi062:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-04-08T19:44:23.500 INFO:teuthology.provision.fog.smithi062:Node is ready 2024-04-08T19:44:23.500 DEBUG:teuthology.orchestra.run.smithi062:> hostname 2024-04-08T19:44:23.515 INFO:teuthology.orchestra.run.smithi062.stdout:smithi062.front.sepia.ceph.com 2024-04-08T19:44:23.515 DEBUG:teuthology.orchestra.run.smithi062:> grep smithi062.front.sepia.ceph.com /etc/hosts 2024-04-08T19:44:23.571 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62 smithi062.front.sepia.ceph.com smithi062 2024-04-08T19:44:23.571 DEBUG:teuthology.orchestra.run.smithi062:> sudo hostname smithi062 2024-04-08T19:44:23.679 DEBUG:teuthology.orchestra.run.smithi062:> sudo sed -i -e 's/smithi062.front.sepia.ceph.com/smithi062/g' /etc/hosts 2024-04-08T19:44:23.711 DEBUG:teuthology.orchestra.run.smithi062:> sudo sed -i -e 's/172.21.15.62/172.21.15.62/g' /etc/hosts 2024-04-08T19:44:23.782 DEBUG:teuthology.orchestra.run.smithi062:> sudo sed -i -e 's/smithi062.front.sepia.ceph.com/smithi062/g' /etc/hostname 2024-04-08T19:44:23.853 DEBUG:teuthology.orchestra.run.smithi062:> sudo hostname smithi062 2024-04-08T19:44:23.924 DEBUG:teuthology.orchestra.run.smithi062:> cat /etc/os-release 2024-04-08T19:44:23.978 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="CentOS Stream" 2024-04-08T19:44:23.979 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="9" 2024-04-08T19:44:23.979 INFO:teuthology.orchestra.run.smithi062.stdout:ID="centos" 2024-04-08T19:44:23.979 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="rhel fedora" 2024-04-08T19:44:23.979 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="9" 2024-04-08T19:44:23.979 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el9" 2024-04-08T19:44:23.979 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-08T19:44:23.979 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31" 2024-04-08T19:44:23.979 INFO:teuthology.orchestra.run.smithi062.stdout:LOGO="fedora-logo-icon" 2024-04-08T19:44:23.979 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-08T19:44:23.979 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://centos.org/" 2024-04-08T19:44:23.979 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-08T19:44:23.979 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-08T19:44:23.979 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-08T19:44:23.980 INFO:teuthology.provision.fog.smithi062:Deploy complete! 2024-04-08T19:44:23.980 DEBUG:teuthology.parallel:result is None 2024-04-08T19:44:24.153 INFO:teuthology.lock.ops:Checking smithi062.front.sepia.ceph.com 2024-04-08T19:44:24.153 INFO:teuthology.lock.ops:New key found. Updating... 2024-04-08T19:44:24.184 INFO:teuthology.lock.ops:Checking smithi087.front.sepia.ceph.com 2024-04-08T19:44:24.185 INFO:teuthology.lock.ops:New key found. Updating... 2024-04-08T19:44:24.211 INFO:teuthology.lock.ops:Updating [smithi062.front.sepia.ceph.com]: set os type and version on server 2024-04-08T19:44:24.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-08T19:44:24.212 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-08T19:44:24.310 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-04-08T19:44:24.377 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-08T19:44:24.377 DEBUG:teuthology.orchestra.run.smithi062:> uname -m 2024-04-08T19:44:24.432 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2024-04-08T19:44:24.432 DEBUG:teuthology.orchestra.run.smithi062:> cat /etc/os-release 2024-04-08T19:44:24.487 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="CentOS Stream" 2024-04-08T19:44:24.487 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="9" 2024-04-08T19:44:24.488 INFO:teuthology.orchestra.run.smithi062.stdout:ID="centos" 2024-04-08T19:44:24.488 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="rhel fedora" 2024-04-08T19:44:24.488 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="9" 2024-04-08T19:44:24.488 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el9" 2024-04-08T19:44:24.488 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-08T19:44:24.488 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31" 2024-04-08T19:44:24.488 INFO:teuthology.orchestra.run.smithi062.stdout:LOGO="fedora-logo-icon" 2024-04-08T19:44:24.488 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-08T19:44:24.488 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://centos.org/" 2024-04-08T19:44:24.488 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-08T19:44:24.488 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-08T19:44:24.488 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-08T19:44:24.489 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2024-04-08T19:44:24.510 INFO:teuthology.lock.ops:Updating [smithi087.front.sepia.ceph.com]: set os type and version on server 2024-04-08T19:44:24.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-04-08T19:44:24.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-08T19:44:24.599 DEBUG:teuthology.orchestra.run.smithi087:> true 2024-04-08T19:44:24.666 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2024-04-08T19:44:24.666 DEBUG:teuthology.orchestra.run.smithi087:> uname -m 2024-04-08T19:44:24.723 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2024-04-08T19:44:24.723 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2024-04-08T19:44:24.779 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="CentOS Stream" 2024-04-08T19:44:24.779 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="9" 2024-04-08T19:44:24.779 INFO:teuthology.orchestra.run.smithi087.stdout:ID="centos" 2024-04-08T19:44:24.779 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE="rhel fedora" 2024-04-08T19:44:24.779 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="9" 2024-04-08T19:44:24.779 INFO:teuthology.orchestra.run.smithi087.stdout:PLATFORM_ID="platform:el9" 2024-04-08T19:44:24.780 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-08T19:44:24.780 INFO:teuthology.orchestra.run.smithi087.stdout:ANSI_COLOR="0;31" 2024-04-08T19:44:24.780 INFO:teuthology.orchestra.run.smithi087.stdout:LOGO="fedora-logo-icon" 2024-04-08T19:44:24.780 INFO:teuthology.orchestra.run.smithi087.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-08T19:44:24.780 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://centos.org/" 2024-04-08T19:44:24.780 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-08T19:44:24.780 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-08T19:44:24.780 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-08T19:44:24.780 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2024-04-08T19:44:24.810 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T19:44:24.971 INFO:teuthology.dispatcher.supervisor:Running job 7645901 2024-04-08T19:44:24.971 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/bin/teuthology -v --owner scheduled_phlogistonjohn@teuthology --archive /home/teuthworker/archive/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645901 --name phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645901/orig.config.yaml 2024-04-08T19:44:24.975 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/phlogistonjohn-2024-04-08_19:15:43-orch:cephadm-wip-phlogistonjohn-testing-2024-04-06-1715-distro-default-smithi/7645901 2024-04-08T19:44:24.975 INFO:teuthology.dispatcher.supervisor:Job PID: 1831936 2024-04-08T19:44:24.975 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2024-04-08T19:46:25.031 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T19:48:25.187 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T19:50:25.356 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T19:52:25.524 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T19:54:25.695 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T19:56:25.851 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T19:58:26.025 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:00:26.193 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:02:26.359 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:04:26.498 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:06:26.666 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:08:26.850 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:10:26.954 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:12:27.145 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:14:27.320 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:16:27.491 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:18:27.655 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:20:27.823 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:22:27.999 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:24:28.159 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:26:28.335 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:28:28.551 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:30:28.724 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:32:28.891 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:34:29.059 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:36:29.231 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:38:29.404 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:40:29.563 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:42:29.731 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:44:29.887 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:46:30.063 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:48:30.227 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:50:30.346 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:52:30.531 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:54:30.667 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:56:30.839 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T20:58:31.014 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:00:31.175 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:02:31.336 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:04:31.507 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:06:31.671 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:08:31.831 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:10:32.008 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:12:32.187 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:14:32.359 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:16:32.471 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:18:32.640 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:20:32.773 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:22:32.937 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:24:33.103 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:26:33.267 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:28:33.438 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:30:33.611 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:32:33.781 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:34:33.955 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:36:34.122 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:38:34.291 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:40:34.463 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:42:34.639 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:44:34.807 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:46:34.975 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:48:35.140 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:50:35.287 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:52:35.421 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:54:35.584 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:56:35.747 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T21:58:35.880 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:00:36.055 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:02:36.204 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:04:36.352 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:06:36.539 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:08:36.715 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:10:36.887 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:12:37.060 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:14:37.227 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:16:37.399 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:18:37.510 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:20:37.642 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:22:37.801 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:24:37.958 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:26:38.122 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:28:38.303 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:30:38.495 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:32:38.667 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:34:38.846 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:36:38.955 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:38:39.080 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:40:39.242 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:42:39.432 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:44:39.544 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:46:39.715 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:48:39.879 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:50:40.058 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:52:40.235 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:54:40.419 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:56:40.595 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T22:58:40.764 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T23:00:40.951 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T23:02:41.123 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T23:04:41.299 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T23:06:41.475 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T23:08:41.639 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T23:10:41.853 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T23:12:42.009 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T23:14:42.186 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T23:16:42.271 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T23:18:42.435 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T23:20:42.611 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T23:22:42.787 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T23:24:42.960 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T23:26:43.103 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T23:28:43.273 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T23:30:43.446 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T23:32:43.615 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-08T23:32:43.676 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1