2024-03-07T13:37:47.477 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-07T13:37:47.477 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T13:37:47.671 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9ea51f3c10>, signals=[15]) 2024-03-07T13:37:47.672 INFO:teuthology.lock.ops:Start node 'smithi008.front.sepia.ceph.com' reimaging 2024-03-07T13:37:47.672 INFO:teuthology.lock.ops:Updating [smithi008.front.sepia.ceph.com]: reset os type and version on server 2024-03-07T13:37:47.672 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2024-03-07T13:37:47.701 INFO:teuthology.lock.ops:Start node 'smithi026.front.sepia.ceph.com' reimaging 2024-03-07T13:37:47.701 INFO:teuthology.lock.ops:Updating [smithi026.front.sepia.ceph.com]: reset os type and version on server 2024-03-07T13:37:47.701 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2024-03-07T13:37:47.732 INFO:teuthology.lock.ops:Start node 'smithi040.front.sepia.ceph.com' reimaging 2024-03-07T13:37:47.732 INFO:teuthology.lock.ops:Updating [smithi040.front.sepia.ceph.com]: reset os type and version on server 2024-03-07T13:37:47.732 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2024-03-07T13:37:47.807 DEBUG:teuthology.provision.fog.smithi008:Requesting image smithi_centos_8.stream (ID 3) 2024-03-07T13:37:47.840 DEBUG:teuthology.provision.fog.smithi026:Requesting image smithi_centos_8.stream (ID 3) 2024-03-07T13:37:47.853 INFO:teuthology.provision.fog.smithi008:Scheduling deploy of centos 8.stream 2024-03-07T13:37:47.874 DEBUG:teuthology.provision.fog.smithi040:Requesting image smithi_centos_8.stream (ID 3) 2024-03-07T13:37:47.891 INFO:teuthology.provision.fog.smithi026:Scheduling deploy of centos 8.stream 2024-03-07T13:37:47.919 INFO:teuthology.provision.fog.smithi040:Scheduling deploy of centos 8.stream 2024-03-07T13:37:48.183 INFO:teuthology.orchestra.console.smithi026:Power off 2024-03-07T13:37:48.183 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-07T13:37:48.195 INFO:teuthology.orchestra.console.smithi008:Power off 2024-03-07T13:37:48.196 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-07T13:37:48.204 DEBUG:teuthology.orchestra.console.smithi026:power off output: Chassis Power Control: Down/Off 2024-03-07T13:37:48.205 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T13:37:48.217 DEBUG:teuthology.orchestra.console.smithi008:power off output: Chassis Power Control: Down/Off 2024-03-07T13:37:48.217 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T13:37:48.230 INFO:teuthology.orchestra.console.smithi040:Power off 2024-03-07T13:37:48.231 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-07T13:37:48.241 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-03-07T13:37:48.246 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is on 2024-03-07T13:37:48.256 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off 2024-03-07T13:37:48.256 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T13:37:48.278 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-03-07T13:37:52.244 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T13:37:52.251 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T13:37:52.279 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T13:37:52.354 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-03-07T13:37:52.361 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is off 2024-03-07T13:37:52.390 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-03-07T13:37:52.462 INFO:teuthology.orchestra.console.smithi008:Power off completed 2024-03-07T13:37:52.563 INFO:teuthology.orchestra.console.smithi008:Power on 2024-03-07T13:37:52.563 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-07T13:37:52.586 DEBUG:teuthology.orchestra.console.smithi008:power on output: Chassis Power Control: Up/On 2024-03-07T13:37:52.587 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T13:37:52.609 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is off 2024-03-07T13:37:56.359 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T13:37:56.392 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T13:37:56.469 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off 2024-03-07T13:37:56.504 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-03-07T13:37:56.571 INFO:teuthology.orchestra.console.smithi026:Power off completed 2024-03-07T13:37:56.610 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T13:37:56.672 INFO:teuthology.orchestra.console.smithi026:Power on 2024-03-07T13:37:56.672 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-07T13:37:56.706 DEBUG:teuthology.orchestra.console.smithi026:power on output: Chassis Power Control: Up/On 2024-03-07T13:37:56.707 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T13:37:56.720 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is off 2024-03-07T13:37:56.735 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off 2024-03-07T13:38:00.508 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T13:38:00.616 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-03-07T13:38:00.717 INFO:teuthology.orchestra.console.smithi040:Power off completed 2024-03-07T13:38:00.721 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T13:38:00.736 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T13:38:00.819 INFO:teuthology.orchestra.console.smithi040:Power on 2024-03-07T13:38:00.819 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-03-07T13:38:00.833 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is on 2024-03-07T13:38:00.843 DEBUG:teuthology.orchestra.console.smithi040:power on output: Chassis Power Control: Up/On 2024-03-07T13:38:00.843 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T13:38:00.852 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off 2024-03-07T13:38:00.866 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-03-07T13:38:00.934 INFO:teuthology.orchestra.console.smithi008:Power on completed 2024-03-07T13:38:01.035 INFO:teuthology.provision.fog.smithi008:Waiting for deploy to finish 2024-03-07T13:38:04.857 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T13:38:04.867 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-07T13:38:04.966 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-03-07T13:38:04.976 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-03-07T13:38:05.067 INFO:teuthology.orchestra.console.smithi026:Power on completed 2024-03-07T13:38:05.077 INFO:teuthology.orchestra.console.smithi040:Power on completed 2024-03-07T13:38:05.168 INFO:teuthology.provision.fog.smithi026:Waiting for deploy to finish 2024-03-07T13:38:05.178 INFO:teuthology.provision.fog.smithi040:Waiting for deploy to finish 2024-03-07T13:40:32.165 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:40:35.228 WARNING:teuthology.provision.fog.smithi008:[Errno None] Unable to connect to port 22 on 172.21.15.8 2024-03-07T13:40:36.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:40:36.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:40:39.327 WARNING:teuthology.provision.fog.smithi040:[Errno None] Unable to connect to port 22 on 172.21.15.40 2024-03-07T13:40:39.360 WARNING:teuthology.provision.fog.smithi026:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-03-07T13:40:41.230 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:40:44.284 WARNING:teuthology.provision.fog.smithi008:[Errno None] Unable to connect to port 22 on 172.21.15.8 2024-03-07T13:40:45.330 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:40:45.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:40:48.412 WARNING:teuthology.provision.fog.smithi040:[Errno None] Unable to connect to port 22 on 172.21.15.40 2024-03-07T13:40:48.412 WARNING:teuthology.provision.fog.smithi026:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-03-07T13:40:50.287 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:40:53.340 WARNING:teuthology.provision.fog.smithi008:[Errno None] Unable to connect to port 22 on 172.21.15.8 2024-03-07T13:40:54.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:40:54.417 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:40:57.468 WARNING:teuthology.provision.fog.smithi040:[Errno None] Unable to connect to port 22 on 172.21.15.40 2024-03-07T13:40:57.468 WARNING:teuthology.provision.fog.smithi026:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-03-07T13:40:59.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:41:02.396 WARNING:teuthology.provision.fog.smithi008:[Errno None] Unable to connect to port 22 on 172.21.15.8 2024-03-07T13:41:03.470 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:41:03.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:41:06.524 WARNING:teuthology.provision.fog.smithi040:[Errno None] Unable to connect to port 22 on 172.21.15.40 2024-03-07T13:41:06.524 WARNING:teuthology.provision.fog.smithi026:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-03-07T13:41:08.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:41:11.452 WARNING:teuthology.provision.fog.smithi008:[Errno None] Unable to connect to port 22 on 172.21.15.8 2024-03-07T13:41:12.526 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:41:12.529 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:41:15.580 WARNING:teuthology.provision.fog.smithi040:[Errno None] Unable to connect to port 22 on 172.21.15.40 2024-03-07T13:41:17.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:41:20.508 WARNING:teuthology.provision.fog.smithi008:[Errno None] Unable to connect to port 22 on 172.21.15.8 2024-03-07T13:41:21.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:41:24.640 WARNING:teuthology.provision.fog.smithi040:[Errno None] Unable to connect to port 22 on 172.21.15.40 2024-03-07T13:41:26.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:41:29.564 WARNING:teuthology.provision.fog.smithi008:[Errno None] Unable to connect to port 22 on 172.21.15.8 2024-03-07T13:41:30.642 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:41:35.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:42:07.436 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-07T13:42:07.437 DEBUG:teuthology.orchestra.run.smithi008:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-07T13:42:12.531 WARNING:teuthology.provision.fog.smithi026:timed out 2024-03-07T13:42:18.533 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:42:18.780 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-07T13:42:18.780 DEBUG:teuthology.orchestra.run.smithi026:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-07T13:42:19.034 INFO:teuthology.provision.fog.smithi026:Node is ready 2024-03-07T13:42:19.034 DEBUG:teuthology.orchestra.run.smithi026:> hostname 2024-03-07T13:42:19.054 INFO:teuthology.orchestra.run.smithi026.stdout:smithi026.front.sepia.ceph.com 2024-03-07T13:42:19.054 DEBUG:teuthology.orchestra.run.smithi026:> grep smithi026.front.sepia.ceph.com /etc/hosts 2024-03-07T13:42:19.114 INFO:teuthology.orchestra.run.smithi026.stdout:172.21.15.26 smithi026.front.sepia.ceph.com smithi026 2024-03-07T13:42:19.115 DEBUG:teuthology.orchestra.run.smithi026:> sudo hostname smithi026 2024-03-07T13:42:19.217 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i -e 's/smithi026.front.sepia.ceph.com/smithi026/g' /etc/hosts 2024-03-07T13:42:19.248 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i -e 's/172.21.15.26/172.21.15.26/g' /etc/hosts 2024-03-07T13:42:19.318 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i -e 's/smithi026.front.sepia.ceph.com/smithi026/g' /etc/hostname 2024-03-07T13:42:19.387 DEBUG:teuthology.orchestra.run.smithi026:> sudo hostname smithi026 2024-03-07T13:42:19.455 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2024-03-07T13:42:19.514 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="CentOS Stream" 2024-03-07T13:42:19.514 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="8" 2024-03-07T13:42:19.514 INFO:teuthology.orchestra.run.smithi026.stdout:ID="centos" 2024-03-07T13:42:19.514 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE="rhel fedora" 2024-03-07T13:42:19.514 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="8" 2024-03-07T13:42:19.514 INFO:teuthology.orchestra.run.smithi026.stdout:PLATFORM_ID="platform:el8" 2024-03-07T13:42:19.515 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-07T13:42:19.515 INFO:teuthology.orchestra.run.smithi026.stdout:ANSI_COLOR="0;31" 2024-03-07T13:42:19.515 INFO:teuthology.orchestra.run.smithi026.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-07T13:42:19.515 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://centos.org/" 2024-03-07T13:42:19.515 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-07T13:42:19.515 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-07T13:42:19.515 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-07T13:42:19.516 INFO:teuthology.provision.fog.smithi026:Deploy complete! 2024-03-07T13:42:19.517 DEBUG:teuthology.parallel:result is None 2024-03-07T13:42:27.078 INFO:teuthology.provision.fog.smithi008:Node is ready 2024-03-07T13:42:27.079 DEBUG:teuthology.orchestra.run.smithi008:> hostname 2024-03-07T13:42:27.103 INFO:teuthology.orchestra.run.smithi008.stdout:smithi008.front.sepia.ceph.com 2024-03-07T13:42:27.104 DEBUG:teuthology.orchestra.run.smithi008:> grep smithi008.front.sepia.ceph.com /etc/hosts 2024-03-07T13:42:27.165 INFO:teuthology.orchestra.run.smithi008.stdout:172.21.15.8 smithi008.front.sepia.ceph.com smithi008 2024-03-07T13:42:27.166 DEBUG:teuthology.orchestra.run.smithi008:> sudo hostname smithi008 2024-03-07T13:42:27.291 DEBUG:teuthology.orchestra.run.smithi008:> sudo sed -i -e 's/smithi008.front.sepia.ceph.com/smithi008/g' /etc/hosts 2024-03-07T13:42:27.323 DEBUG:teuthology.orchestra.run.smithi008:> sudo sed -i -e 's/172.21.15.8/172.21.15.8/g' /etc/hosts 2024-03-07T13:42:27.393 DEBUG:teuthology.orchestra.run.smithi008:> sudo sed -i -e 's/smithi008.front.sepia.ceph.com/smithi008/g' /etc/hostname 2024-03-07T13:42:27.464 DEBUG:teuthology.orchestra.run.smithi008:> sudo hostname smithi008 2024-03-07T13:42:27.534 DEBUG:teuthology.orchestra.run.smithi008:> cat /etc/os-release 2024-03-07T13:42:27.594 INFO:teuthology.orchestra.run.smithi008.stdout:NAME="CentOS Stream" 2024-03-07T13:42:27.594 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION="8" 2024-03-07T13:42:27.594 INFO:teuthology.orchestra.run.smithi008.stdout:ID="centos" 2024-03-07T13:42:27.594 INFO:teuthology.orchestra.run.smithi008.stdout:ID_LIKE="rhel fedora" 2024-03-07T13:42:27.594 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_ID="8" 2024-03-07T13:42:27.594 INFO:teuthology.orchestra.run.smithi008.stdout:PLATFORM_ID="platform:el8" 2024-03-07T13:42:27.594 INFO:teuthology.orchestra.run.smithi008.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-07T13:42:27.594 INFO:teuthology.orchestra.run.smithi008.stdout:ANSI_COLOR="0;31" 2024-03-07T13:42:27.594 INFO:teuthology.orchestra.run.smithi008.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-07T13:42:27.594 INFO:teuthology.orchestra.run.smithi008.stdout:HOME_URL="https://centos.org/" 2024-03-07T13:42:27.594 INFO:teuthology.orchestra.run.smithi008.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-07T13:42:27.595 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-07T13:42:27.595 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-07T13:42:27.595 INFO:teuthology.provision.fog.smithi008:Deploy complete! 2024-03-07T13:42:27.595 DEBUG:teuthology.parallel:result is None 2024-03-07T13:42:30.648 WARNING:teuthology.provision.fog.smithi040:timed out 2024-03-07T13:42:36.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:42:36.917 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-03-07T13:42:36.917 DEBUG:teuthology.orchestra.run.smithi040:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-03-07T13:42:37.149 INFO:teuthology.provision.fog.smithi040:Node is ready 2024-03-07T13:42:37.149 DEBUG:teuthology.orchestra.run.smithi040:> hostname 2024-03-07T13:42:37.209 INFO:teuthology.orchestra.run.smithi040.stdout:smithi040.front.sepia.ceph.com 2024-03-07T13:42:37.209 DEBUG:teuthology.orchestra.run.smithi040:> grep smithi040.front.sepia.ceph.com /etc/hosts 2024-03-07T13:42:37.268 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40 smithi040.front.sepia.ceph.com smithi040 2024-03-07T13:42:37.269 DEBUG:teuthology.orchestra.run.smithi040:> sudo hostname smithi040 2024-03-07T13:42:37.373 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i -e 's/smithi040.front.sepia.ceph.com/smithi040/g' /etc/hosts 2024-03-07T13:42:37.406 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i -e 's/172.21.15.40/172.21.15.40/g' /etc/hosts 2024-03-07T13:42:37.475 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i -e 's/smithi040.front.sepia.ceph.com/smithi040/g' /etc/hostname 2024-03-07T13:42:37.545 DEBUG:teuthology.orchestra.run.smithi040:> sudo hostname smithi040 2024-03-07T13:42:37.614 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2024-03-07T13:42:37.673 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2024-03-07T13:42:37.673 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="8" 2024-03-07T13:42:37.673 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2024-03-07T13:42:37.673 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2024-03-07T13:42:37.673 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="8" 2024-03-07T13:42:37.673 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el8" 2024-03-07T13:42:37.673 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-07T13:42:37.673 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2024-03-07T13:42:37.673 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-07T13:42:37.673 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2024-03-07T13:42:37.673 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-07T13:42:37.673 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-07T13:42:37.674 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-07T13:42:37.674 INFO:teuthology.provision.fog.smithi040:Deploy complete! 2024-03-07T13:42:37.674 DEBUG:teuthology.parallel:result is None 2024-03-07T13:42:37.871 INFO:teuthology.lock.ops:Checking smithi008.front.sepia.ceph.com 2024-03-07T13:42:37.871 INFO:teuthology.lock.ops:New key found. Updating... 2024-03-07T13:42:37.891 INFO:teuthology.lock.ops:Checking smithi026.front.sepia.ceph.com 2024-03-07T13:42:37.891 INFO:teuthology.lock.ops:New key found. Updating... 2024-03-07T13:42:37.908 INFO:teuthology.lock.ops:Checking smithi040.front.sepia.ceph.com 2024-03-07T13:42:37.908 INFO:teuthology.lock.ops:New key found. Updating... 2024-03-07T13:42:37.926 INFO:teuthology.lock.ops:Updating [smithi008.front.sepia.ceph.com]: set os type and version on server 2024-03-07T13:42:37.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-03-07T13:42:37.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:42:38.002 DEBUG:teuthology.orchestra.run.smithi008:> true 2024-03-07T13:42:38.140 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-03-07T13:42:38.140 DEBUG:teuthology.orchestra.run.smithi008:> uname -m 2024-03-07T13:42:38.200 INFO:teuthology.orchestra.run.smithi008.stdout:x86_64 2024-03-07T13:42:38.200 DEBUG:teuthology.orchestra.run.smithi008:> cat /etc/os-release 2024-03-07T13:42:38.260 INFO:teuthology.orchestra.run.smithi008.stdout:NAME="CentOS Stream" 2024-03-07T13:42:38.260 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION="8" 2024-03-07T13:42:38.260 INFO:teuthology.orchestra.run.smithi008.stdout:ID="centos" 2024-03-07T13:42:38.260 INFO:teuthology.orchestra.run.smithi008.stdout:ID_LIKE="rhel fedora" 2024-03-07T13:42:38.260 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_ID="8" 2024-03-07T13:42:38.260 INFO:teuthology.orchestra.run.smithi008.stdout:PLATFORM_ID="platform:el8" 2024-03-07T13:42:38.261 INFO:teuthology.orchestra.run.smithi008.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-07T13:42:38.261 INFO:teuthology.orchestra.run.smithi008.stdout:ANSI_COLOR="0;31" 2024-03-07T13:42:38.261 INFO:teuthology.orchestra.run.smithi008.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-07T13:42:38.261 INFO:teuthology.orchestra.run.smithi008.stdout:HOME_URL="https://centos.org/" 2024-03-07T13:42:38.261 INFO:teuthology.orchestra.run.smithi008.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-07T13:42:38.261 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-07T13:42:38.261 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-07T13:42:38.261 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2024-03-07T13:42:38.283 INFO:teuthology.lock.ops:Updating [smithi026.front.sepia.ceph.com]: set os type and version on server 2024-03-07T13:42:38.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-07T13:42:38.284 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:42:38.358 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-03-07T13:42:38.500 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-07T13:42:38.500 DEBUG:teuthology.orchestra.run.smithi026:> uname -m 2024-03-07T13:42:38.560 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2024-03-07T13:42:38.560 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2024-03-07T13:42:38.621 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="CentOS Stream" 2024-03-07T13:42:38.621 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="8" 2024-03-07T13:42:38.622 INFO:teuthology.orchestra.run.smithi026.stdout:ID="centos" 2024-03-07T13:42:38.622 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE="rhel fedora" 2024-03-07T13:42:38.622 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="8" 2024-03-07T13:42:38.622 INFO:teuthology.orchestra.run.smithi026.stdout:PLATFORM_ID="platform:el8" 2024-03-07T13:42:38.622 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-07T13:42:38.623 INFO:teuthology.orchestra.run.smithi026.stdout:ANSI_COLOR="0;31" 2024-03-07T13:42:38.623 INFO:teuthology.orchestra.run.smithi026.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-07T13:42:38.623 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://centos.org/" 2024-03-07T13:42:38.623 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-07T13:42:38.623 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-07T13:42:38.623 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-07T13:42:38.624 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2024-03-07T13:42:38.649 INFO:teuthology.lock.ops:Updating [smithi040.front.sepia.ceph.com]: set os type and version on server 2024-03-07T13:42:38.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-03-07T13:42:38.650 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-07T13:42:38.734 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-03-07T13:42:38.806 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-03-07T13:42:38.806 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2024-03-07T13:42:38.868 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2024-03-07T13:42:38.868 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2024-03-07T13:42:38.929 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2024-03-07T13:42:38.929 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="8" 2024-03-07T13:42:38.929 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2024-03-07T13:42:38.929 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2024-03-07T13:42:38.929 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="8" 2024-03-07T13:42:38.929 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el8" 2024-03-07T13:42:38.929 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 8" 2024-03-07T13:42:38.929 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2024-03-07T13:42:38.930 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-03-07T13:42:38.930 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2024-03-07T13:42:38.930 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-07T13:42:38.930 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-07T13:42:38.930 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-07T13:42:38.930 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2024-03-07T13:42:38.979 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T13:42:39.172 INFO:teuthology.dispatcher.supervisor:Running job 7584489 2024-03-07T13:42:39.172 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7584489 --name vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi --description fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} -- /home/teuthworker/archive/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7584489/orig.config.yaml 2024-03-07T13:42:39.176 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/7584489 2024-03-07T13:42:39.176 INFO:teuthology.dispatcher.supervisor:Job PID: 693302 2024-03-07T13:42:39.176 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2024-03-07T13:44:39.273 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T13:46:39.441 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T13:48:39.614 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T13:49:39.650 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 504: 504 Gateway Time-out

504 Gateway Time-out


nginx/1.18.0 (Ubuntu)
2024-03-07T13:50:36.992 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: 502 Bad Gateway

502 Bad Gateway


nginx/1.18.0 (Ubuntu)
2024-03-07T13:50:59.626 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: 502 Bad Gateway

502 Bad Gateway


nginx/1.18.0 (Ubuntu)
2024-03-07T13:51:03.209 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: 502 Bad Gateway

502 Bad Gateway


nginx/1.18.0 (Ubuntu)
2024-03-07T13:51:07.665 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: 502 Bad Gateway

502 Bad Gateway


nginx/1.18.0 (Ubuntu)
2024-03-07T13:51:12.989 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: 502 Bad Gateway

502 Bad Gateway


nginx/1.18.0 (Ubuntu)
2024-03-07T13:51:19.150 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: 502 Bad Gateway

502 Bad Gateway


nginx/1.18.0 (Ubuntu)
2024-03-07T13:51:26.171 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: 502 Bad Gateway

502 Bad Gateway


nginx/1.18.0 (Ubuntu)
2024-03-07T13:51:34.050 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: 502 Bad Gateway

502 Bad Gateway


nginx/1.18.0 (Ubuntu)
2024-03-07T13:51:42.786 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/vshankar-2024-03-07_10:41:12-fs-wip-vshankar-testing-20240307.022905-testing-default-smithi/jobs/ failed with status 502: 502 Bad Gateway

502 Bad Gateway


nginx/1.18.0 (Ubuntu)
2024-03-07T13:51:42.786 ERROR:teuthology.dispatcher.supervisor:Failed to report job status; ignoring Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/dispatcher/supervisor.py", line 335, in run_with_watchdog report.try_push_job_info(job_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/report.py", line 512, in try_push_job_info push_job_info(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/report.py", line 474, in push_job_info reporter.report_job(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/report.py", line 299, in report_job while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'report job 7584489' reached maximum tries (11) after waiting for 48.63960229248602 seconds 2024-03-07T13:53:42.876 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-07T13:53:42.963 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1