2023-08-08T11:37:10.035 INFO:root:teuthology version: 0.0.1.dev181+g7fda959 2023-08-08T11:37:10.037 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-08T11:37:10.194 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1207caef70>, signals=[15]) 2023-08-08T11:37:10.197 INFO:teuthology.lock.ops:Start node 'smithi072.front.sepia.ceph.com' reimaging 2023-08-08T11:37:10.197 INFO:teuthology.lock.ops:Updating [smithi072.front.sepia.ceph.com]: reset os type and version on server 2023-08-08T11:37:10.198 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2023-08-08T11:37:10.239 INFO:teuthology.lock.ops:Start node 'smithi121.front.sepia.ceph.com' reimaging 2023-08-08T11:37:10.239 INFO:teuthology.lock.ops:Updating [smithi121.front.sepia.ceph.com]: reset os type and version on server 2023-08-08T11:37:10.240 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2023-08-08T11:37:10.242 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /host {"name": "smithi072"} 2023-08-08T11:37:10.263 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /host {"name": "smithi121"} 2023-08-08T11:37:10.324 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /image {"name": "smithi_rhel_8.6"} 2023-08-08T11:37:10.332 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /image {"name": "smithi_rhel_8.6"} 2023-08-08T11:37:10.374 DEBUG:teuthology.provision.fog.smithi121:Requesting image smithi_rhel_8.6 (ID 5) 2023-08-08T11:37:10.374 DEBUG:teuthology.provision.fog.smithi121:FOG request: PUT /host/226 {"imageID": 5} 2023-08-08T11:37:10.378 DEBUG:teuthology.provision.fog.smithi072:Requesting image smithi_rhel_8.6 (ID 5) 2023-08-08T11:37:10.378 DEBUG:teuthology.provision.fog.smithi072:FOG request: PUT /host/177 {"imageID": 5} 2023-08-08T11:37:10.415 INFO:teuthology.provision.fog.smithi121:Scheduling deploy of rhel 8.6 2023-08-08T11:37:10.415 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-08T11:37:10.423 INFO:teuthology.provision.fog.smithi072:Scheduling deploy of rhel 8.6 2023-08-08T11:37:10.423 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /task/active None 2023-08-08T11:37:10.466 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /tasktype {"name": "deploy"} 2023-08-08T11:37:10.471 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /tasktype {"name": "deploy"} 2023-08-08T11:37:10.503 DEBUG:teuthology.provision.fog.smithi121:FOG request: POST /host/226/task {"taskTypeID": 1} 2023-08-08T11:37:10.506 DEBUG:teuthology.provision.fog.smithi072:FOG request: POST /host/177/task {"taskTypeID": 1} 2023-08-08T11:37:10.562 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-08T11:37:10.565 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /task/active None 2023-08-08T11:37:10.651 INFO:teuthology.orchestra.console.smithi072:Power off 2023-08-08T11:37:10.652 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-08T11:37:10.665 INFO:teuthology.orchestra.console.smithi121:Power off 2023-08-08T11:37:10.666 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-08T11:37:10.675 DEBUG:teuthology.orchestra.console.smithi072:power off output: Chassis Power Control: Down/Off 2023-08-08T11:37:10.675 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-08T11:37:10.686 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2023-08-08T11:37:10.686 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-08T11:37:10.699 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2023-08-08T11:37:10.708 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2023-08-08T11:37:14.700 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-08T11:37:14.709 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-08T11:37:14.812 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2023-08-08T11:37:14.822 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2023-08-08T11:37:18.816 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-08T11:37:18.825 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-08T11:37:18.927 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2023-08-08T11:37:18.937 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2023-08-08T11:37:22.932 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-08T11:37:22.941 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-08T11:37:23.044 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is off 2023-08-08T11:37:23.053 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2023-08-08T11:37:23.145 INFO:teuthology.orchestra.console.smithi072:Power off completed 2023-08-08T11:37:23.154 INFO:teuthology.orchestra.console.smithi121:Power off completed 2023-08-08T11:37:23.247 INFO:teuthology.orchestra.console.smithi072:Power on 2023-08-08T11:37:23.248 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-08-08T11:37:23.256 INFO:teuthology.orchestra.console.smithi121:Power on 2023-08-08T11:37:23.257 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-08-08T11:37:23.269 DEBUG:teuthology.orchestra.console.smithi072:power on output: Chassis Power Control: Up/On 2023-08-08T11:37:23.270 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-08T11:37:23.280 DEBUG:teuthology.orchestra.console.smithi121:power on output: Chassis Power Control: Up/On 2023-08-08T11:37:23.280 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-08T11:37:23.292 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is off 2023-08-08T11:37:23.301 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2023-08-08T11:37:27.296 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-08T11:37:27.304 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-08T11:37:27.406 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2023-08-08T11:37:27.415 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2023-08-08T11:37:27.508 INFO:teuthology.orchestra.console.smithi072:Power on completed 2023-08-08T11:37:27.609 INFO:teuthology.provision.fog.smithi072:Waiting for deploy to finish 2023-08-08T11:37:27.610 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /task/active None 2023-08-08T11:37:31.418 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-08T11:37:31.527 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2023-08-08T11:37:31.628 INFO:teuthology.orchestra.console.smithi121:Power on completed 2023-08-08T11:37:31.729 INFO:teuthology.provision.fog.smithi121:Waiting for deploy to finish 2023-08-08T11:37:31.729 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-08T11:37:42.672 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /task/active None 2023-08-08T11:37:46.792 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-08T11:37:57.736 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /task/active None 2023-08-08T11:38:01.859 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-08T11:38:12.808 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /task/active None 2023-08-08T11:38:16.920 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-08T11:38:27.874 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /task/active None 2023-08-08T11:38:32.000 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-08T11:38:42.948 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /task/active None 2023-08-08T11:38:47.072 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-08T11:38:58.020 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /task/active None 2023-08-08T11:39:02.148 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-08T11:39:13.084 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /task/active None 2023-08-08T11:39:17.216 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-08T11:39:28.160 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /task/active None 2023-08-08T11:39:32.289 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-08T11:39:43.236 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /task/active None 2023-08-08T11:39:47.360 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-08T11:39:58.312 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /task/active None 2023-08-08T11:40:02.425 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-08T11:40:13.380 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /task/active None 2023-08-08T11:40:17.488 DEBUG:teuthology.provision.fog.smithi121:FOG request: GET /task/active None 2023-08-08T11:40:17.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:40:20.632 WARNING:teuthology.provision.fog.smithi121:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-08-08T11:40:26.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:40:28.443 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /task/active None 2023-08-08T11:40:29.712 WARNING:teuthology.provision.fog.smithi121:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-08-08T11:40:35.716 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:40:38.772 WARNING:teuthology.provision.fog.smithi121:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-08-08T11:40:43.516 DEBUG:teuthology.provision.fog.smithi072:FOG request: GET /task/active None 2023-08-08T11:40:43.585 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:40:44.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:40:46.640 WARNING:teuthology.provision.fog.smithi072:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-08-08T11:40:47.856 WARNING:teuthology.provision.fog.smithi121:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-08-08T11:40:52.644 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:40:53.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:40:55.700 WARNING:teuthology.provision.fog.smithi072:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-08-08T11:40:56.912 WARNING:teuthology.provision.fog.smithi121:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-08-08T11:41:01.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:41:02.915 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:41:04.784 WARNING:teuthology.provision.fog.smithi072:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-08-08T11:41:05.968 WARNING:teuthology.provision.fog.smithi121:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-08-08T11:41:10.788 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:41:11.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:41:13.844 WARNING:teuthology.provision.fog.smithi072:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-08-08T11:41:19.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:41:22.936 WARNING:teuthology.provision.fog.smithi072:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-08-08T11:41:28.940 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:41:32.016 WARNING:teuthology.provision.fog.smithi072:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-08-08T11:41:38.020 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:41:41.072 WARNING:teuthology.provision.fog.smithi072:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-08-08T11:41:47.076 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:42:11.996 WARNING:teuthology.provision.fog.smithi121:timed out 2023-08-08T11:42:18.002 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:42:18.376 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-08-08T11:42:18.377 DEBUG:teuthology.orchestra.run.smithi121:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-08-08T11:42:18.642 INFO:teuthology.provision.fog.smithi121:Node is ready 2023-08-08T11:42:18.643 DEBUG:teuthology.orchestra.run.smithi121:> hostname 2023-08-08T11:42:18.703 INFO:teuthology.orchestra.run.smithi121.stdout:smithi121 2023-08-08T11:42:18.704 DEBUG:teuthology.orchestra.run.smithi121:> grep smithi121 /etc/hosts 2023-08-08T11:42:18.764 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121 smithi121.front.sepia.ceph.com smithi121 2023-08-08T11:42:18.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo hostname smithi121 2023-08-08T11:42:18.879 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i -e 's/smithi121/smithi121/g' /etc/hosts 2023-08-08T11:42:18.906 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i -e 's/172.21.15.121/172.21.15.121/g' /etc/hosts 2023-08-08T11:42:18.976 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i -e 's/smithi121/smithi121/g' /etc/hostname 2023-08-08T11:42:19.046 DEBUG:teuthology.orchestra.run.smithi121:> sudo hostname smithi121 2023-08-08T11:42:19.115 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2023-08-08T11:42:19.175 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="Red Hat Enterprise Linux" 2023-08-08T11:42:19.176 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="8.6 (Ootpa)" 2023-08-08T11:42:19.176 INFO:teuthology.orchestra.run.smithi121.stdout:ID="rhel" 2023-08-08T11:42:19.176 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="fedora" 2023-08-08T11:42:19.176 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="8.6" 2023-08-08T11:42:19.176 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el8" 2023-08-08T11:42:19.177 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-08-08T11:42:19.177 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2023-08-08T11:42:19.177 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-08-08T11:42:19.177 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://www.redhat.com/" 2023-08-08T11:42:19.177 INFO:teuthology.orchestra.run.smithi121.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-08-08T11:42:19.177 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-08T11:42:19.177 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-08-08T11:42:19.178 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-08-08T11:42:19.178 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-08-08T11:42:19.178 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-08-08T11:42:19.178 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-08-08T11:42:19.179 INFO:teuthology.provision.fog.smithi121:Deploy complete! 2023-08-08T11:42:19.180 DEBUG:teuthology.parallel:result is None 2023-08-08T11:42:47.104 WARNING:teuthology.provision.fog.smithi072:timed out 2023-08-08T11:42:53.111 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:42:53.438 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-08-08T11:42:53.438 DEBUG:teuthology.orchestra.run.smithi072:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-08-08T11:42:53.795 INFO:teuthology.provision.fog.smithi072:Node is ready 2023-08-08T11:42:53.796 DEBUG:teuthology.orchestra.run.smithi072:> hostname 2023-08-08T11:42:53.815 INFO:teuthology.orchestra.run.smithi072.stdout:smithi072 2023-08-08T11:42:53.815 DEBUG:teuthology.orchestra.run.smithi072:> grep smithi072 /etc/hosts 2023-08-08T11:42:53.875 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72 smithi072.front.sepia.ceph.com smithi072 2023-08-08T11:42:53.876 DEBUG:teuthology.orchestra.run.smithi072:> sudo hostname smithi072 2023-08-08T11:42:54.020 DEBUG:teuthology.orchestra.run.smithi072:> sudo sed -i -e 's/smithi072/smithi072/g' /etc/hosts 2023-08-08T11:42:54.049 DEBUG:teuthology.orchestra.run.smithi072:> sudo sed -i -e 's/172.21.15.72/172.21.15.72/g' /etc/hosts 2023-08-08T11:42:54.118 DEBUG:teuthology.orchestra.run.smithi072:> sudo sed -i -e 's/smithi072/smithi072/g' /etc/hostname 2023-08-08T11:42:54.186 DEBUG:teuthology.orchestra.run.smithi072:> sudo hostname smithi072 2023-08-08T11:42:54.253 DEBUG:teuthology.orchestra.run.smithi072:> cat /etc/os-release 2023-08-08T11:42:54.309 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="Red Hat Enterprise Linux" 2023-08-08T11:42:54.309 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="8.6 (Ootpa)" 2023-08-08T11:42:54.309 INFO:teuthology.orchestra.run.smithi072.stdout:ID="rhel" 2023-08-08T11:42:54.309 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="fedora" 2023-08-08T11:42:54.310 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="8.6" 2023-08-08T11:42:54.310 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el8" 2023-08-08T11:42:54.310 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-08-08T11:42:54.310 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31" 2023-08-08T11:42:54.310 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-08-08T11:42:54.310 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://www.redhat.com/" 2023-08-08T11:42:54.310 INFO:teuthology.orchestra.run.smithi072.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-08-08T11:42:54.311 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-08T11:42:54.311 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-08-08T11:42:54.311 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-08-08T11:42:54.311 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-08-08T11:42:54.311 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-08-08T11:42:54.311 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-08-08T11:42:54.311 INFO:teuthology.provision.fog.smithi072:Deploy complete! 2023-08-08T11:42:54.312 DEBUG:teuthology.parallel:result is None 2023-08-08T11:42:54.481 INFO:teuthology.lock.ops:Checking smithi072.front.sepia.ceph.com 2023-08-08T11:42:54.481 INFO:teuthology.lock.ops:New key found. Updating... 2023-08-08T11:42:54.506 INFO:teuthology.lock.ops:Checking smithi121.front.sepia.ceph.com 2023-08-08T11:42:54.507 INFO:teuthology.lock.ops:New key found. Updating... 2023-08-08T11:42:54.533 INFO:teuthology.lock.ops:Updating [smithi072.front.sepia.ceph.com]: set os type and version on server 2023-08-08T11:42:54.533 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-08-08T11:42:54.534 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:42:54.598 DEBUG:teuthology.orchestra.run.smithi072:> true 2023-08-08T11:42:54.664 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-08-08T11:42:54.664 DEBUG:teuthology.orchestra.run.smithi072:> uname -m 2023-08-08T11:42:54.721 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64 2023-08-08T11:42:54.721 DEBUG:teuthology.orchestra.run.smithi072:> cat /etc/os-release 2023-08-08T11:42:54.778 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="Red Hat Enterprise Linux" 2023-08-08T11:42:54.778 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="8.6 (Ootpa)" 2023-08-08T11:42:54.778 INFO:teuthology.orchestra.run.smithi072.stdout:ID="rhel" 2023-08-08T11:42:54.778 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="fedora" 2023-08-08T11:42:54.778 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="8.6" 2023-08-08T11:42:54.779 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el8" 2023-08-08T11:42:54.779 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-08-08T11:42:54.779 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31" 2023-08-08T11:42:54.779 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-08-08T11:42:54.779 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://www.redhat.com/" 2023-08-08T11:42:54.779 INFO:teuthology.orchestra.run.smithi072.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-08-08T11:42:54.779 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-08T11:42:54.780 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-08-08T11:42:54.780 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-08-08T11:42:54.780 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-08-08T11:42:54.780 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-08-08T11:42:54.780 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-08-08T11:42:54.782 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2023-08-08T11:42:54.798 INFO:teuthology.lock.ops:Updating [smithi121.front.sepia.ceph.com]: set os type and version on server 2023-08-08T11:42:54.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-08-08T11:42:54.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-08T11:42:54.863 DEBUG:teuthology.orchestra.run.smithi121:> true 2023-08-08T11:42:54.993 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-08-08T11:42:54.994 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2023-08-08T11:42:55.050 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2023-08-08T11:42:55.051 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2023-08-08T11:42:55.107 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="Red Hat Enterprise Linux" 2023-08-08T11:42:55.108 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="8.6 (Ootpa)" 2023-08-08T11:42:55.108 INFO:teuthology.orchestra.run.smithi121.stdout:ID="rhel" 2023-08-08T11:42:55.108 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="fedora" 2023-08-08T11:42:55.109 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="8.6" 2023-08-08T11:42:55.109 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el8" 2023-08-08T11:42:55.109 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-08-08T11:42:55.109 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2023-08-08T11:42:55.109 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-08-08T11:42:55.110 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://www.redhat.com/" 2023-08-08T11:42:55.110 INFO:teuthology.orchestra.run.smithi121.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-08-08T11:42:55.110 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-08T11:42:55.110 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-08-08T11:42:55.110 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-08-08T11:42:55.110 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-08-08T11:42:55.111 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-08-08T11:42:55.111 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-08-08T11:42:55.111 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2023-08-08T11:42:55.132 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-08T11:42:55.239 INFO:teuthology.dispatcher.supervisor:Running job 7362579 2023-08-08T11:42:55.240 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_7fda95956ac10132c9b74016ba832db907df09fa/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2023-08-08_11:25:00-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7362579 --name vshankar-2023-08-08_11:25:00-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} -- /home/teuthworker/archive/vshankar-2023-08-08_11:25:00-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7362579/orig.config.yaml 2023-08-08T11:42:55.243 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2023-08-08_11:25:00-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7362579 2023-08-08T11:42:55.244 INFO:teuthology.dispatcher.supervisor:Job PID: 3811 2023-08-08T11:42:55.244 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-08-08T11:44:55.344 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-08T11:46:55.517 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-08T11:48:55.656 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-08T11:50:55.825 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-08T11:52:56.017 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-08T11:54:56.185 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-08T11:56:56.361 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-08T11:58:56.533 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-08T12:00:56.697 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-08T12:02:56.861 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-08T12:02:56.924 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120 2023-08-08T12:02:57.032 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2023-08-08T12:02:57.033 INFO:teuthology.nuke:Checking targets against current locks 2023-08-08T12:02:57.078 DEBUG:teuthology.nuke:shortname: smithi072 2023-08-08T12:02:57.079 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-08T12:02:57.092 DEBUG:teuthology.nuke:shortname: smithi121 2023-08-08T12:02:57.092 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-08T12:02:57.099 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-08-08_11:25:00-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7362579', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-08 11:37:09.197203', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-08-08T12:02:57.111 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-08-08_11:25:00-fs-wip-vshankar-testing-20230808.093601-testing-default-smithi/7362579', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-08 11:37:09.195131', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-08-08T12:02:57.140 INFO:teuthology.orchestra.console.smithi072:Power off 2023-08-08T12:02:57.141 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-08T12:02:57.165 DEBUG:teuthology.orchestra.console.smithi072:power off output: Chassis Power Control: Down/Off 2023-08-08T12:02:57.166 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-08T12:02:57.183 INFO:teuthology.orchestra.console.smithi121:Power off 2023-08-08T12:02:57.184 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-08T12:02:57.193 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is off 2023-08-08T12:02:57.203 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2023-08-08T12:02:57.204 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-08T12:02:57.227 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2023-08-08T12:02:57.294 INFO:teuthology.orchestra.console.smithi072:Power off completed 2023-08-08T12:02:57.329 INFO:teuthology.orchestra.console.smithi121:Power off completed 2023-08-08T12:02:57.447 INFO:teuthology.lock.ops:unlocked: smithi072.front.sepia.ceph.com 2023-08-08T12:02:57.481 INFO:teuthology.lock.ops:unlocked: smithi121.front.sepia.ceph.com