2024-04-01T16:55:45.628 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-01T16:55:45.629 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T16:55:45.807 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f19ad711c10>, signals=[15]) 2024-04-01T16:55:45.807 INFO:teuthology.lock.ops:Start node 'smithi049.front.sepia.ceph.com' reimaging 2024-04-01T16:55:45.808 INFO:teuthology.lock.ops:Updating [smithi049.front.sepia.ceph.com]: reset os type and version on server 2024-04-01T16:55:45.808 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2024-04-01T16:55:45.831 INFO:teuthology.lock.ops:Start node 'smithi053.front.sepia.ceph.com' reimaging 2024-04-01T16:55:45.831 INFO:teuthology.lock.ops:Updating [smithi053.front.sepia.ceph.com]: reset os type and version on server 2024-04-01T16:55:45.831 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-04-01T16:55:45.855 INFO:teuthology.lock.ops:Start node 'smithi107.front.sepia.ceph.com' reimaging 2024-04-01T16:55:45.856 INFO:teuthology.lock.ops:Updating [smithi107.front.sepia.ceph.com]: reset os type and version on server 2024-04-01T16:55:45.856 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-04-01T16:55:45.944 DEBUG:teuthology.provision.fog.smithi049:Requesting image smithi_centos_9.stream (ID 10) 2024-04-01T16:55:45.970 DEBUG:teuthology.provision.fog.smithi053:Requesting image smithi_centos_9.stream (ID 10) 2024-04-01T16:55:45.990 INFO:teuthology.provision.fog.smithi049:Scheduling deploy of centos 9.stream 2024-04-01T16:55:45.993 DEBUG:teuthology.provision.fog.smithi107:Requesting image smithi_centos_9.stream (ID 10) 2024-04-01T16:55:46.020 INFO:teuthology.provision.fog.smithi053:Scheduling deploy of centos 9.stream 2024-04-01T16:55:46.033 INFO:teuthology.provision.fog.smithi107:Scheduling deploy of centos 9.stream 2024-04-01T16:55:46.214 INFO:teuthology.orchestra.console.smithi049:Power off 2024-04-01T16:55:46.215 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-01T16:55:46.241 DEBUG:teuthology.orchestra.console.smithi049:power off output: Chassis Power Control: Down/Off 2024-04-01T16:55:46.241 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T16:55:46.258 INFO:teuthology.orchestra.console.smithi053:Power off 2024-04-01T16:55:46.258 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-01T16:55:46.272 INFO:teuthology.orchestra.console.smithi107:Power off 2024-04-01T16:55:46.272 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-01T16:55:46.279 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2024-04-01T16:55:46.280 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2024-04-01T16:55:46.281 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T16:55:46.296 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2024-04-01T16:55:46.296 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T16:55:46.304 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-04-01T16:55:46.319 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-04-01T16:55:46.406 INFO:teuthology.orchestra.console.smithi053:Power off completed 2024-04-01T16:55:46.507 INFO:teuthology.orchestra.console.smithi053:Power on 2024-04-01T16:55:46.507 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-01T16:55:46.528 DEBUG:teuthology.orchestra.console.smithi053:power on output: Chassis Power Control: Up/On 2024-04-01T16:55:46.529 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T16:55:46.550 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-04-01T16:55:50.282 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T16:55:50.320 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T16:55:50.392 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2024-04-01T16:55:50.431 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-04-01T16:55:50.551 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T16:55:50.660 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-04-01T16:55:54.394 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T16:55:54.432 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T16:55:54.503 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2024-04-01T16:55:54.541 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2024-04-01T16:55:54.642 INFO:teuthology.orchestra.console.smithi107:Power off completed 2024-04-01T16:55:54.661 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T16:55:54.743 INFO:teuthology.orchestra.console.smithi107:Power on 2024-04-01T16:55:54.743 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-01T16:55:54.763 DEBUG:teuthology.orchestra.console.smithi107:power on output: Chassis Power Control: Up/On 2024-04-01T16:55:54.764 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T16:55:54.773 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-04-01T16:55:54.785 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2024-04-01T16:55:54.874 INFO:teuthology.orchestra.console.smithi053:Power on completed 2024-04-01T16:55:54.975 INFO:teuthology.provision.fog.smithi053:Waiting for deploy to finish 2024-04-01T16:55:58.506 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T16:55:58.615 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is off 2024-04-01T16:55:58.716 INFO:teuthology.orchestra.console.smithi049:Power off completed 2024-04-01T16:55:58.786 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T16:55:58.817 INFO:teuthology.orchestra.console.smithi049:Power on 2024-04-01T16:55:58.817 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2024-04-01T16:55:58.838 DEBUG:teuthology.orchestra.console.smithi049:power on output: Chassis Power Control: Up/On 2024-04-01T16:55:58.839 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T16:55:58.859 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is off 2024-04-01T16:55:58.894 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2024-04-01T16:56:02.862 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T16:56:02.896 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T16:56:02.971 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is off 2024-04-01T16:56:03.006 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-04-01T16:56:03.106 INFO:teuthology.orchestra.console.smithi107:Power on completed 2024-04-01T16:56:03.207 INFO:teuthology.provision.fog.smithi107:Waiting for deploy to finish 2024-04-01T16:56:06.975 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T16:56:07.084 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2024-04-01T16:56:07.185 INFO:teuthology.orchestra.console.smithi049:Power on completed 2024-04-01T16:56:07.286 INFO:teuthology.provision.fog.smithi049:Waiting for deploy to finish 2024-04-01T16:58:10.541 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T16:58:29.022 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-04-01T16:58:35.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T16:58:38.078 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-04-01T16:58:44.085 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T16:58:47.166 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-04-01T16:58:48.970 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T16:58:52.030 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-01T16:58:53.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T16:58:56.226 WARNING:teuthology.provision.fog.smithi053:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-04-01T16:58:58.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T16:59:01.086 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-01T16:59:02.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T16:59:07.090 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T16:59:10.146 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-01T16:59:16.150 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T16:59:19.230 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-01T16:59:25.234 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T16:59:28.286 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-01T16:59:34.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T16:59:37.342 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-04-01T16:59:38.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T16:59:41.246 WARNING:teuthology.provision.fog.smithi049:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-01T16:59:43.346 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T16:59:47.251 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T16:59:50.306 WARNING:teuthology.provision.fog.smithi049:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-01T16:59:56.311 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T16:59:59.390 WARNING:teuthology.provision.fog.smithi049:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-01T17:00:02.233 WARNING:teuthology.provision.fog.smithi053:timed out 2024-04-01T17:00:05.394 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T17:00:08.235 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T17:00:08.446 WARNING:teuthology.provision.fog.smithi049:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-01T17:00:08.485 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-04-01T17:00:08.486 DEBUG:teuthology.orchestra.run.smithi053:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-04-01T17:00:08.825 INFO:teuthology.provision.fog.smithi053:Node is ready 2024-04-01T17:00:08.825 DEBUG:teuthology.orchestra.run.smithi053:> hostname 2024-04-01T17:00:08.841 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053.front.sepia.ceph.com 2024-04-01T17:00:08.841 DEBUG:teuthology.orchestra.run.smithi053:> grep smithi053.front.sepia.ceph.com /etc/hosts 2024-04-01T17:00:08.897 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53 smithi053.front.sepia.ceph.com smithi053 2024-04-01T17:00:08.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo hostname smithi053 2024-04-01T17:00:09.023 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i -e 's/smithi053.front.sepia.ceph.com/smithi053/g' /etc/hosts 2024-04-01T17:00:09.056 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i -e 's/172.21.15.53/172.21.15.53/g' /etc/hosts 2024-04-01T17:00:09.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i -e 's/smithi053.front.sepia.ceph.com/smithi053/g' /etc/hostname 2024-04-01T17:00:09.199 DEBUG:teuthology.orchestra.run.smithi053:> sudo hostname smithi053 2024-04-01T17:00:09.270 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-04-01T17:00:09.326 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2024-04-01T17:00:09.326 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="9" 2024-04-01T17:00:09.326 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2024-04-01T17:00:09.326 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2024-04-01T17:00:09.326 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="9" 2024-04-01T17:00:09.326 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el9" 2024-04-01T17:00:09.326 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T17:00:09.327 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2024-04-01T17:00:09.327 INFO:teuthology.orchestra.run.smithi053.stdout:LOGO="fedora-logo-icon" 2024-04-01T17:00:09.327 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T17:00:09.327 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2024-04-01T17:00:09.327 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T17:00:09.327 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T17:00:09.327 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T17:00:09.328 INFO:teuthology.provision.fog.smithi053:Deploy complete! 2024-04-01T17:00:09.329 DEBUG:teuthology.parallel:result is None 2024-04-01T17:00:14.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T17:00:17.502 WARNING:teuthology.provision.fog.smithi049:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-01T17:00:23.507 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T17:00:26.558 WARNING:teuthology.provision.fog.smithi049:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-01T17:00:32.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T17:00:35.614 WARNING:teuthology.provision.fog.smithi049:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-01T17:00:41.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T17:00:43.350 WARNING:teuthology.provision.fog.smithi107:timed out 2024-04-01T17:00:44.702 WARNING:teuthology.provision.fog.smithi049:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-01T17:00:49.357 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T17:00:49.644 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-04-01T17:00:49.644 DEBUG:teuthology.orchestra.run.smithi107:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-04-01T17:00:49.952 INFO:teuthology.provision.fog.smithi107:Node is ready 2024-04-01T17:00:49.952 DEBUG:teuthology.orchestra.run.smithi107:> hostname 2024-04-01T17:00:49.967 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107.front.sepia.ceph.com 2024-04-01T17:00:49.967 DEBUG:teuthology.orchestra.run.smithi107:> grep smithi107.front.sepia.ceph.com /etc/hosts 2024-04-01T17:00:50.021 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107 smithi107.front.sepia.ceph.com smithi107 2024-04-01T17:00:50.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname smithi107 2024-04-01T17:00:50.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i -e 's/smithi107.front.sepia.ceph.com/smithi107/g' /etc/hosts 2024-04-01T17:00:50.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i -e 's/172.21.15.107/172.21.15.107/g' /etc/hosts 2024-04-01T17:00:50.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i -e 's/smithi107.front.sepia.ceph.com/smithi107/g' /etc/hostname 2024-04-01T17:00:50.293 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname smithi107 2024-04-01T17:00:50.360 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-04-01T17:00:50.414 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-04-01T17:00:50.414 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2024-04-01T17:00:50.414 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-04-01T17:00:50.414 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-04-01T17:00:50.414 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2024-04-01T17:00:50.415 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2024-04-01T17:00:50.415 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T17:00:50.415 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-04-01T17:00:50.415 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2024-04-01T17:00:50.415 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T17:00:50.415 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-04-01T17:00:50.415 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T17:00:50.415 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T17:00:50.415 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T17:00:50.415 INFO:teuthology.provision.fog.smithi107:Deploy complete! 2024-04-01T17:00:50.416 DEBUG:teuthology.parallel:result is None 2024-04-01T17:00:50.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T17:00:53.762 WARNING:teuthology.provision.fog.smithi049:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-01T17:00:59.766 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T17:01:02.846 WARNING:teuthology.provision.fog.smithi049:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-01T17:01:08.850 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T17:01:11.906 WARNING:teuthology.provision.fog.smithi049:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-04-01T17:01:17.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T17:02:17.966 WARNING:teuthology.provision.fog.smithi049:timed out 2024-04-01T17:02:23.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T17:02:24.268 INFO:teuthology.orchestra.run:Running command with timeout 600 2024-04-01T17:02:24.268 DEBUG:teuthology.orchestra.run.smithi049:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2024-04-01T17:02:24.613 INFO:teuthology.provision.fog.smithi049:Node is ready 2024-04-01T17:02:24.614 DEBUG:teuthology.orchestra.run.smithi049:> hostname 2024-04-01T17:02:24.629 INFO:teuthology.orchestra.run.smithi049.stdout:smithi049.front.sepia.ceph.com 2024-04-01T17:02:24.629 DEBUG:teuthology.orchestra.run.smithi049:> grep smithi049.front.sepia.ceph.com /etc/hosts 2024-04-01T17:02:24.685 INFO:teuthology.orchestra.run.smithi049.stdout:172.21.15.49 smithi049.front.sepia.ceph.com smithi049 2024-04-01T17:02:24.686 DEBUG:teuthology.orchestra.run.smithi049:> sudo hostname smithi049 2024-04-01T17:02:24.802 DEBUG:teuthology.orchestra.run.smithi049:> sudo sed -i -e 's/smithi049.front.sepia.ceph.com/smithi049/g' /etc/hosts 2024-04-01T17:02:24.835 DEBUG:teuthology.orchestra.run.smithi049:> sudo sed -i -e 's/172.21.15.49/172.21.15.49/g' /etc/hosts 2024-04-01T17:02:24.906 DEBUG:teuthology.orchestra.run.smithi049:> sudo sed -i -e 's/smithi049.front.sepia.ceph.com/smithi049/g' /etc/hostname 2024-04-01T17:02:24.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo hostname smithi049 2024-04-01T17:02:25.047 DEBUG:teuthology.orchestra.run.smithi049:> cat /etc/os-release 2024-04-01T17:02:25.102 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="CentOS Stream" 2024-04-01T17:02:25.103 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="9" 2024-04-01T17:02:25.103 INFO:teuthology.orchestra.run.smithi049.stdout:ID="centos" 2024-04-01T17:02:25.103 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="rhel fedora" 2024-04-01T17:02:25.103 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="9" 2024-04-01T17:02:25.103 INFO:teuthology.orchestra.run.smithi049.stdout:PLATFORM_ID="platform:el9" 2024-04-01T17:02:25.103 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T17:02:25.103 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31" 2024-04-01T17:02:25.103 INFO:teuthology.orchestra.run.smithi049.stdout:LOGO="fedora-logo-icon" 2024-04-01T17:02:25.103 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T17:02:25.103 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://centos.org/" 2024-04-01T17:02:25.103 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T17:02:25.103 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T17:02:25.103 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T17:02:25.104 INFO:teuthology.provision.fog.smithi049:Deploy complete! 2024-04-01T17:02:25.104 DEBUG:teuthology.parallel:result is None 2024-04-01T17:02:25.360 INFO:teuthology.lock.ops:Checking smithi049.front.sepia.ceph.com 2024-04-01T17:02:25.360 INFO:teuthology.lock.ops:New key found. Updating... 2024-04-01T17:02:25.391 INFO:teuthology.lock.ops:Checking smithi053.front.sepia.ceph.com 2024-04-01T17:02:25.391 INFO:teuthology.lock.ops:New key found. Updating... 2024-04-01T17:02:25.415 INFO:teuthology.lock.ops:Checking smithi107.front.sepia.ceph.com 2024-04-01T17:02:25.415 INFO:teuthology.lock.ops:New key found. Updating... 2024-04-01T17:02:25.434 INFO:teuthology.lock.ops:Updating [smithi049.front.sepia.ceph.com]: set os type and version on server 2024-04-01T17:02:25.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-01T17:02:25.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T17:02:25.536 DEBUG:teuthology.orchestra.run.smithi049:> true 2024-04-01T17:02:25.600 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-04-01T17:02:25.600 DEBUG:teuthology.orchestra.run.smithi049:> uname -m 2024-04-01T17:02:25.655 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2024-04-01T17:02:25.656 DEBUG:teuthology.orchestra.run.smithi049:> cat /etc/os-release 2024-04-01T17:02:25.710 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="CentOS Stream" 2024-04-01T17:02:25.710 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="9" 2024-04-01T17:02:25.710 INFO:teuthology.orchestra.run.smithi049.stdout:ID="centos" 2024-04-01T17:02:25.710 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="rhel fedora" 2024-04-01T17:02:25.710 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="9" 2024-04-01T17:02:25.710 INFO:teuthology.orchestra.run.smithi049.stdout:PLATFORM_ID="platform:el9" 2024-04-01T17:02:25.711 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T17:02:25.711 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31" 2024-04-01T17:02:25.711 INFO:teuthology.orchestra.run.smithi049.stdout:LOGO="fedora-logo-icon" 2024-04-01T17:02:25.711 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T17:02:25.711 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://centos.org/" 2024-04-01T17:02:25.711 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T17:02:25.711 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T17:02:25.711 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T17:02:25.711 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2024-04-01T17:02:25.740 INFO:teuthology.lock.ops:Updating [smithi053.front.sepia.ceph.com]: set os type and version on server 2024-04-01T17:02:25.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-04-01T17:02:25.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T17:02:25.843 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-04-01T17:02:26.029 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-04-01T17:02:26.029 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2024-04-01T17:02:26.084 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2024-04-01T17:02:26.085 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-04-01T17:02:26.139 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2024-04-01T17:02:26.139 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="9" 2024-04-01T17:02:26.140 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2024-04-01T17:02:26.140 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2024-04-01T17:02:26.140 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="9" 2024-04-01T17:02:26.140 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el9" 2024-04-01T17:02:26.140 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T17:02:26.140 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2024-04-01T17:02:26.140 INFO:teuthology.orchestra.run.smithi053.stdout:LOGO="fedora-logo-icon" 2024-04-01T17:02:26.140 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T17:02:26.140 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2024-04-01T17:02:26.140 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T17:02:26.140 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T17:02:26.140 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T17:02:26.141 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-04-01T17:02:26.274 INFO:teuthology.lock.ops:Updating [smithi107.front.sepia.ceph.com]: set os type and version on server 2024-04-01T17:02:26.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-04-01T17:02:26.275 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T17:02:26.364 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-04-01T17:02:26.537 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-04-01T17:02:26.537 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-04-01T17:02:26.592 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-04-01T17:02:26.592 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-04-01T17:02:26.647 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-04-01T17:02:26.648 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2024-04-01T17:02:26.648 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-04-01T17:02:26.648 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-04-01T17:02:26.648 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2024-04-01T17:02:26.648 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2024-04-01T17:02:26.648 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T17:02:26.648 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-04-01T17:02:26.648 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2024-04-01T17:02:26.648 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T17:02:26.648 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-04-01T17:02:26.648 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T17:02:26.648 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T17:02:26.648 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T17:02:26.649 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-04-01T17:02:26.673 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T17:02:26.766 INFO:teuthology.dispatcher.supervisor:Running job 7634001 2024-04-01T17:02:26.766 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2024-04-01_14:11:32-fs-wip-vshankar-testing-20240330.172700-testing-default-smithi/7634001 --name vshankar-2024-04-01_14:11:32-fs-wip-vshankar-testing-20240330.172700-testing-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} -- /home/teuthworker/archive/vshankar-2024-04-01_14:11:32-fs-wip-vshankar-testing-20240330.172700-testing-default-smithi/7634001/orig.config.yaml 2024-04-01T17:02:26.769 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2024-04-01_14:11:32-fs-wip-vshankar-testing-20240330.172700-testing-default-smithi/7634001 2024-04-01T17:02:26.770 INFO:teuthology.dispatcher.supervisor:Job PID: 3553678 2024-04-01T17:02:26.770 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2024-04-01T17:04:26.871 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T17:06:27.035 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T17:08:27.199 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T17:10:27.366 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T17:12:27.540 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T17:14:27.715 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T17:16:27.874 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T17:18:28.047 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T17:20:28.259 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T17:20:28.331 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1