2023-07-24T11:29:22.728 INFO:root:teuthology version: 0.0.1.dev158+g1426592
2023-07-24T11:29:22.729 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T11:29:22.730 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T11:29:22.873 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7feb7442cd60>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7feb73ddb310>, signals=[15])
2023-07-24T11:29:22.874 INFO:teuthology.lock.ops:Start node 'smithi122.front.sepia.ceph.com' reimaging
2023-07-24T11:29:22.874 INFO:teuthology.lock.ops:Updating [smithi122.front.sepia.ceph.com]: reset os type and version on server
2023-07-24T11:29:22.875 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server
2023-07-24T11:29:22.903 INFO:teuthology.lock.ops:Start node 'smithi203.front.sepia.ceph.com' reimaging
2023-07-24T11:29:22.903 INFO:teuthology.lock.ops:Updating [smithi203.front.sepia.ceph.com]: reset os type and version on server
2023-07-24T11:29:22.903 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server
2023-07-24T11:29:22.904 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /host {"name": "smithi122"}
2023-07-24T11:29:22.929 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /host {"name": "smithi203"}
2023-07-24T11:29:22.984 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-07-24T11:29:23.000 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-07-24T11:29:23.026 DEBUG:teuthology.provision.fog.smithi122:Requesting image smithi_rhel_8.6 (ID 5)
2023-07-24T11:29:23.027 DEBUG:teuthology.provision.fog.smithi122:FOG request: PUT /host/227 {"imageID": 5}
2023-07-24T11:29:23.044 DEBUG:teuthology.provision.fog.smithi203:Requesting image smithi_rhel_8.6 (ID 5)
2023-07-24T11:29:23.044 DEBUG:teuthology.provision.fog.smithi203:FOG request: PUT /host/307 {"imageID": 5}
2023-07-24T11:29:23.067 INFO:teuthology.provision.fog.smithi122:Scheduling deploy of rhel 8.6
2023-07-24T11:29:23.068 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:29:23.083 INFO:teuthology.provision.fog.smithi203:Scheduling deploy of rhel 8.6
2023-07-24T11:29:23.083 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:29:23.112 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /tasktype {"name": "deploy"}
2023-07-24T11:29:23.119 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /tasktype {"name": "deploy"}
2023-07-24T11:29:23.156 DEBUG:teuthology.provision.fog.smithi122:FOG request: POST /host/227/task {"taskTypeID": 1}
2023-07-24T11:29:23.160 DEBUG:teuthology.provision.fog.smithi203:FOG request: POST /host/307/task {"taskTypeID": 1}
2023-07-24T11:29:23.228 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:29:23.243 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:29:23.310 INFO:teuthology.orchestra.console:Power off smithi122
2023-07-24T11:29:23.311 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-24T11:29:23.327 INFO:teuthology.orchestra.console:Power off smithi203
2023-07-24T11:29:23.328 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-24T11:29:23.339 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off
2023-07-24T11:29:23.340 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-24T11:29:23.352 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off
2023-07-24T11:29:23.353 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-24T11:29:23.364 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-24T11:29:23.380 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-24T11:29:27.368 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-24T11:29:27.382 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-24T11:29:27.479 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-24T11:29:27.495 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-24T11:29:31.485 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-24T11:29:31.497 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-24T11:29:31.597 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-24T11:29:31.610 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-24T11:29:31.699 INFO:teuthology.orchestra.console:Power off for smithi122 completed
2023-07-24T11:29:31.711 INFO:teuthology.orchestra.console:Power off for smithi203 completed
2023-07-24T11:29:31.800 INFO:teuthology.orchestra.console:Power on smithi122
2023-07-24T11:29:31.800 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-24T11:29:31.813 INFO:teuthology.orchestra.console:Power on smithi203
2023-07-24T11:29:31.814 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-24T11:29:31.828 DEBUG:teuthology.orchestra.console:power on output: Chassis Power Control: Up/On
2023-07-24T11:29:31.829 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-24T11:29:31.845 DEBUG:teuthology.orchestra.console:power on output: Chassis Power Control: Up/On
2023-07-24T11:29:31.846 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-24T11:29:31.857 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-24T11:29:31.868 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-24T11:29:35.862 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-24T11:29:35.875 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-24T11:29:35.977 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-24T11:29:35.987 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-24T11:29:36.080 INFO:teuthology.orchestra.console:Power on for smithi122 completed
2023-07-24T11:29:36.181 INFO:teuthology.provision.fog.smithi122:Waiting for deploy to finish
2023-07-24T11:29:36.182 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:29:39.991 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-24T11:29:40.101 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is on
2023-07-24T11:29:40.202 INFO:teuthology.orchestra.console:Power on for smithi203 completed
2023-07-24T11:29:40.303 INFO:teuthology.provision.fog.smithi203:Waiting for deploy to finish
2023-07-24T11:29:40.303 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:29:51.364 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:29:55.448 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:30:06.533 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:30:10.584 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:30:21.688 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:30:25.720 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:30:36.836 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:30:40.862 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:30:51.976 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:30:56.004 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:31:07.172 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:31:11.151 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:31:22.348 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:31:26.320 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:31:37.564 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:31:41.500 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:31:52.776 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:31:56.708 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:32:07.944 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:32:11.876 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:32:23.119 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:32:27.057 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:32:38.293 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:32:42.224 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:32:53.471 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:32:57.481 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:33:08.644 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:33:12.676 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:33:23.816 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:33:27.860 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:33:38.980 DEBUG:teuthology.provision.fog.smithi122:FOG request: GET /task/active None
2023-07-24T11:33:39.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T11:33:43.028 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:33:57.488 WARNING:teuthology.provision.fog.smithi122:[Errno None] Unable to connect to port 22 on 172.21.15.122
2023-07-24T11:33:58.196 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:34:03.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T11:34:06.544 WARNING:teuthology.provision.fog.smithi122:[Errno None] Unable to connect to port 22 on 172.21.15.122
2023-07-24T11:34:12.551 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T11:34:13.338 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:34:15.632 WARNING:teuthology.provision.fog.smithi122:[Errno None] Unable to connect to port 22 on 172.21.15.122
2023-07-24T11:34:21.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T11:34:24.688 WARNING:teuthology.provision.fog.smithi122:[Errno None] Unable to connect to port 22 on 172.21.15.122
2023-07-24T11:34:28.501 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:34:30.692 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T11:34:33.744 WARNING:teuthology.provision.fog.smithi122:[Errno None] Unable to connect to port 22 on 172.21.15.122
2023-07-24T11:34:39.748 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T11:34:43.678 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:34:58.868 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:35:14.048 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:35:29.236 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:35:39.750 WARNING:teuthology.provision.fog.smithi122:timed out
2023-07-24T11:35:44.431 DEBUG:teuthology.provision.fog.smithi203:FOG request: GET /task/active None
2023-07-24T11:35:44.606 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T11:35:45.754 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T11:35:46.075 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-07-24T11:35:46.076 DEBUG:teuthology.orchestra.run.smithi122:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-07-24T11:35:46.378 INFO:teuthology.provision.fog.smithi122:Node is ready
2023-07-24T11:35:46.378 DEBUG:teuthology.orchestra.run.smithi122:> hostname
2023-07-24T11:35:46.397 INFO:teuthology.orchestra.run.smithi122.stdout:smithi122
2023-07-24T11:35:46.398 DEBUG:teuthology.orchestra.run.smithi122:> grep smithi122 /etc/hosts
2023-07-24T11:35:46.457 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.15.122 smithi122.front.sepia.ceph.com smithi122
2023-07-24T11:35:46.457 DEBUG:teuthology.orchestra.run.smithi122:> sudo hostname smithi122
2023-07-24T11:35:46.581 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i -e 's/smithi122/smithi122/g' /etc/hosts
2023-07-24T11:35:46.609 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i -e 's/172.21.15.122/172.21.15.122/g' /etc/hosts
2023-07-24T11:35:46.677 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i -e 's/smithi122/smithi122/g' /etc/hostname
2023-07-24T11:35:46.745 DEBUG:teuthology.orchestra.run.smithi122:> sudo hostname smithi122
2023-07-24T11:35:46.811 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release
2023-07-24T11:35:46.868 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="Red Hat Enterprise Linux"
2023-07-24T11:35:46.868 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="8.6 (Ootpa)"
2023-07-24T11:35:46.868 INFO:teuthology.orchestra.run.smithi122.stdout:ID="rhel"
2023-07-24T11:35:46.868 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="fedora"
2023-07-24T11:35:46.869 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="8.6"
2023-07-24T11:35:46.869 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el8"
2023-07-24T11:35:46.869 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-07-24T11:35:46.869 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31"
2023-07-24T11:35:46.869 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-07-24T11:35:46.869 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://www.redhat.com/"
2023-07-24T11:35:46.869 INFO:teuthology.orchestra.run.smithi122.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-07-24T11:35:46.870 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-07-24T11:35:46.870 INFO:teuthology.orchestra.run.smithi122.stdout:
2023-07-24T11:35:46.870 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-07-24T11:35:46.870 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-07-24T11:35:46.870 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-07-24T11:35:46.870 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-07-24T11:35:46.871 INFO:teuthology.provision.fog.smithi122:Deploy complete!
2023-07-24T11:35:46.872 DEBUG:teuthology.parallel:result is None
2023-07-24T11:36:02.928 WARNING:teuthology.provision.fog.smithi203:[Errno None] Unable to connect to port 22 on 172.21.15.203
2023-07-24T11:36:08.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T11:36:12.016 WARNING:teuthology.provision.fog.smithi203:[Errno None] Unable to connect to port 22 on 172.21.15.203
2023-07-24T11:36:18.017 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T11:36:21.072 WARNING:teuthology.provision.fog.smithi203:[Errno None] Unable to connect to port 22 on 172.21.15.203
2023-07-24T11:36:27.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T11:36:30.128 WARNING:teuthology.provision.fog.smithi203:[Errno None] Unable to connect to port 22 on 172.21.15.203
2023-07-24T11:36:36.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T11:36:39.184 WARNING:teuthology.provision.fog.smithi203:[Errno None] Unable to connect to port 22 on 172.21.15.203
2023-07-24T11:36:45.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T11:37:45.228 WARNING:teuthology.provision.fog.smithi203:timed out
2023-07-24T11:37:51.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T11:37:51.559 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-07-24T11:37:51.559 DEBUG:teuthology.orchestra.run.smithi203:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-07-24T11:37:51.860 INFO:teuthology.provision.fog.smithi203:Node is ready
2023-07-24T11:37:51.860 DEBUG:teuthology.orchestra.run.smithi203:> hostname
2023-07-24T11:37:51.877 INFO:teuthology.orchestra.run.smithi203.stdout:smithi203.front.sepia.ceph.com
2023-07-24T11:37:51.878 DEBUG:teuthology.orchestra.run.smithi203:> grep smithi203.front.sepia.ceph.com /etc/hosts
2023-07-24T11:37:51.935 INFO:teuthology.orchestra.run.smithi203.stdout:172.21.15.203 smithi203.front.sepia.ceph.com smithi203
2023-07-24T11:37:51.935 DEBUG:teuthology.orchestra.run.smithi203:> sudo hostname smithi203
2023-07-24T11:37:52.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i -e 's/smithi203.front.sepia.ceph.com/smithi203/g' /etc/hosts
2023-07-24T11:37:52.076 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i -e 's/172.21.15.203/172.21.15.203/g' /etc/hosts
2023-07-24T11:37:52.143 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i -e 's/smithi203.front.sepia.ceph.com/smithi203/g' /etc/hostname
2023-07-24T11:37:52.211 DEBUG:teuthology.orchestra.run.smithi203:> sudo hostname smithi203
2023-07-24T11:37:52.278 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release
2023-07-24T11:37:52.339 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="Red Hat Enterprise Linux"
2023-07-24T11:37:52.339 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="8.6 (Ootpa)"
2023-07-24T11:37:52.339 INFO:teuthology.orchestra.run.smithi203.stdout:ID="rhel"
2023-07-24T11:37:52.340 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="fedora"
2023-07-24T11:37:52.340 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="8.6"
2023-07-24T11:37:52.340 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el8"
2023-07-24T11:37:52.340 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-07-24T11:37:52.340 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31"
2023-07-24T11:37:52.341 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-07-24T11:37:52.341 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://www.redhat.com/"
2023-07-24T11:37:52.341 INFO:teuthology.orchestra.run.smithi203.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-07-24T11:37:52.341 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-07-24T11:37:52.341 INFO:teuthology.orchestra.run.smithi203.stdout:
2023-07-24T11:37:52.341 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-07-24T11:37:52.341 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-07-24T11:37:52.342 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-07-24T11:37:52.342 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-07-24T11:37:52.342 INFO:teuthology.provision.fog.smithi203:Deploy complete!
2023-07-24T11:37:52.342 DEBUG:teuthology.parallel:result is None
2023-07-24T11:37:52.523 INFO:teuthology.lock.ops:Checking smithi122.front.sepia.ceph.com
2023-07-24T11:37:52.524 INFO:teuthology.lock.ops:New key found. Updating...
2023-07-24T11:37:52.572 INFO:teuthology.lock.ops:Checking smithi203.front.sepia.ceph.com
2023-07-24T11:37:52.573 INFO:teuthology.lock.ops:New key found. Updating...
2023-07-24T11:37:52.626 INFO:teuthology.lock.ops:Updating [smithi122.front.sepia.ceph.com]: set os type and version on server
2023-07-24T11:37:52.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com'
2023-07-24T11:37:52.628 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T11:37:52.693 DEBUG:teuthology.orchestra.run.smithi122:> true
2023-07-24T11:37:52.830 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com'
2023-07-24T11:37:52.830 DEBUG:teuthology.orchestra.run.smithi122:> uname -m
2023-07-24T11:37:52.887 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64
2023-07-24T11:37:52.887 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release
2023-07-24T11:37:52.945 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="Red Hat Enterprise Linux"
2023-07-24T11:37:52.945 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="8.6 (Ootpa)"
2023-07-24T11:37:52.945 INFO:teuthology.orchestra.run.smithi122.stdout:ID="rhel"
2023-07-24T11:37:52.946 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="fedora"
2023-07-24T11:37:52.946 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="8.6"
2023-07-24T11:37:52.946 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el8"
2023-07-24T11:37:52.946 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-07-24T11:37:52.946 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31"
2023-07-24T11:37:52.946 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-07-24T11:37:52.946 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://www.redhat.com/"
2023-07-24T11:37:52.947 INFO:teuthology.orchestra.run.smithi122.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-07-24T11:37:52.947 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-07-24T11:37:52.947 INFO:teuthology.orchestra.run.smithi122.stdout:
2023-07-24T11:37:52.947 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-07-24T11:37:52.947 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-07-24T11:37:52.947 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-07-24T11:37:52.947 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-07-24T11:37:52.948 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server
2023-07-24T11:37:52.972 INFO:teuthology.lock.ops:Updating [smithi203.front.sepia.ceph.com]: set os type and version on server
2023-07-24T11:37:52.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com'
2023-07-24T11:37:52.973 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-24T11:37:53.040 DEBUG:teuthology.orchestra.run.smithi203:> true
2023-07-24T11:37:53.108 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com'
2023-07-24T11:37:53.109 DEBUG:teuthology.orchestra.run.smithi203:> uname -m
2023-07-24T11:37:53.168 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64
2023-07-24T11:37:53.169 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release
2023-07-24T11:37:53.228 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="Red Hat Enterprise Linux"
2023-07-24T11:37:53.229 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="8.6 (Ootpa)"
2023-07-24T11:37:53.229 INFO:teuthology.orchestra.run.smithi203.stdout:ID="rhel"
2023-07-24T11:37:53.229 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="fedora"
2023-07-24T11:37:53.229 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="8.6"
2023-07-24T11:37:53.229 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el8"
2023-07-24T11:37:53.230 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-07-24T11:37:53.230 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31"
2023-07-24T11:37:53.230 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-07-24T11:37:53.230 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://www.redhat.com/"
2023-07-24T11:37:53.230 INFO:teuthology.orchestra.run.smithi203.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-07-24T11:37:53.230 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-07-24T11:37:53.230 INFO:teuthology.orchestra.run.smithi203.stdout:
2023-07-24T11:37:53.231 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-07-24T11:37:53.231 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-07-24T11:37:53.231 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-07-24T11:37:53.231 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-07-24T11:37:53.232 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server
2023-07-24T11:37:53.255 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T11:37:53.256 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T11:37:53.355 INFO:teuthology.dispatcher.supervisor:Running job 7349523
2023-07-24T11:37:53.356 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_1426592083662fef3e03bbeec33624331f0fdbc1/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2023-07-24_11:23:42-rados:thrash-wip-rf-sam2-sched-distro-default-smithi/7349523 --name rfriedma-2023-07-24_11:23:42-rados:thrash-wip-rf-sam2-sched-distro-default-smithi --description rados:thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-snaps-balanced} -- /home/teuthworker/archive/rfriedma-2023-07-24_11:23:42-rados:thrash-wip-rf-sam2-sched-distro-default-smithi/7349523/orig.config.yaml
2023-07-24T11:37:53.359 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2023-07-24_11:23:42-rados:thrash-wip-rf-sam2-sched-distro-default-smithi/7349523
2023-07-24T11:37:53.360 INFO:teuthology.dispatcher.supervisor:Job PID: 12730
2023-07-24T11:37:53.360 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-07-24T11:39:53.460 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T11:39:53.462 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T11:41:53.635 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T11:41:53.636 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T11:43:53.809 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T11:43:53.809 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T11:45:53.987 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T11:45:53.989 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T11:47:54.189 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T11:47:54.190 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T11:49:54.393 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T11:49:54.394 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T11:51:54.581 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T11:51:54.582 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T11:53:54.753 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T11:53:54.754 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T11:55:54.922 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T11:55:54.923 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T11:57:55.093 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T11:57:55.094 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T11:59:55.273 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T11:59:55.274 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T12:01:55.441 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-24T12:01:55.442 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-24T12:01:55.509 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120
2023-07-24T12:01:55.650 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-07-24T12:01:55.650 INFO:teuthology.nuke:Checking targets against current locks
2023-07-24T12:01:55.717 DEBUG:teuthology.nuke:shortname: smithi122
2023-07-24T12:01:55.717 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-24T12:01:55.739 DEBUG:teuthology.nuke:shortname: smithi203
2023-07-24T12:01:55.739 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-24T12:01:55.745 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rfriedma-2023-07-24_11:23:42-rados:thrash-wip-rf-sam2-sched-distro-default-smithi/7349523', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-24 11:29:21.925207', 'locked_by': 'scheduled_rfriedma@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='}
2023-07-24T12:01:55.762 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rfriedma-2023-07-24_11:23:42-rados:thrash-wip-rf-sam2-sched-distro-default-smithi/7349523', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-24 11:29:21.923357', 'locked_by': 'scheduled_rfriedma@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='}
2023-07-24T12:01:55.797 INFO:teuthology.orchestra.console:Power off smithi122
2023-07-24T12:01:55.798 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-24T12:01:55.820 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off
2023-07-24T12:01:55.821 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-24T12:01:55.839 INFO:teuthology.orchestra.console:Power off smithi203
2023-07-24T12:01:55.839 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-24T12:01:55.850 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-24T12:01:55.862 DEBUG:teuthology.orchestra.console:power off output: Chassis Power Control: Down/Off
2023-07-24T12:01:55.863 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-24T12:01:55.886 DEBUG:teuthology.orchestra.console:check power output: Chassis Power is off
2023-07-24T12:01:55.951 INFO:teuthology.orchestra.console:Power off for smithi122 completed
2023-07-24T12:01:55.988 INFO:teuthology.orchestra.console:Power off for smithi203 completed
2023-07-24T12:01:56.108 INFO:teuthology.lock.ops:unlocked: smithi122.front.sepia.ceph.com
2023-07-24T12:01:56.132 INFO:teuthology.lock.ops:unlocked: smithi203.front.sepia.ceph.com