2023-07-15T02:30:17.238 INFO:root:teuthology version: 0.0.1.dev147+gff9991b
2023-07-15T02:30:17.240 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-15T02:30:17.240 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-15T02:30:17.406 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f3d81f6d610>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f3d8318a8b0>, signals=[15])
2023-07-15T02:30:17.408 INFO:teuthology.lock.ops:Start node 'smithi031.front.sepia.ceph.com' reimaging
2023-07-15T02:30:17.408 INFO:teuthology.lock.ops:Updating [smithi031.front.sepia.ceph.com]: reset os type and version on server
2023-07-15T02:30:17.409 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server
2023-07-15T02:30:17.436 INFO:teuthology.lock.ops:Start node 'smithi037.front.sepia.ceph.com' reimaging
2023-07-15T02:30:17.436 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: reset os type and version on server
2023-07-15T02:30:17.436 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server
2023-07-15T02:30:17.437 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /host {"name": "smithi031"}
2023-07-15T02:30:17.461 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /host {"name": "smithi037"}
2023-07-15T02:30:17.519 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-07-15T02:30:17.531 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /image {"name": "smithi_rhel_8.6"}
2023-07-15T02:30:17.562 DEBUG:teuthology.provision.fog.smithi031:Requesting image smithi_rhel_8.6 (ID 5)
2023-07-15T02:30:17.562 DEBUG:teuthology.provision.fog.smithi031:FOG request: PUT /host/136 {"imageID": 5}
2023-07-15T02:30:17.576 DEBUG:teuthology.provision.fog.smithi037:Requesting image smithi_rhel_8.6 (ID 5)
2023-07-15T02:30:17.576 DEBUG:teuthology.provision.fog.smithi037:FOG request: PUT /host/142 {"imageID": 5}
2023-07-15T02:30:17.604 INFO:teuthology.provision.fog.smithi031:Scheduling deploy of rhel 8.6
2023-07-15T02:30:17.604 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-07-15T02:30:17.624 INFO:teuthology.provision.fog.smithi037:Scheduling deploy of rhel 8.6
2023-07-15T02:30:17.624 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /task/active None
2023-07-15T02:30:17.668 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /tasktype {"name": "deploy"}
2023-07-15T02:30:17.691 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /tasktype {"name": "deploy"}
2023-07-15T02:30:17.698 DEBUG:teuthology.provision.fog.smithi031:FOG request: POST /host/136/task {"taskTypeID": 1}
2023-07-15T02:30:17.730 DEBUG:teuthology.provision.fog.smithi037:FOG request: POST /host/142/task {"taskTypeID": 1}
2023-07-15T02:30:17.751 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-07-15T02:30:17.775 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /task/active None
2023-07-15T02:30:17.841 INFO:teuthology.orchestra.console:Power off smithi031
2023-07-15T02:30:17.842 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-15T02:30:17.858 INFO:teuthology.orchestra.console:Power off smithi037
2023-07-15T02:30:17.858 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-15T02:30:17.869 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-15T02:30:17.881 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-15T02:30:21.892 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-15T02:30:21.902 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-15T02:30:26.008 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-15T02:30:26.016 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-15T02:30:26.219 INFO:teuthology.orchestra.console:Power off for smithi031 completed
2023-07-15T02:30:26.320 INFO:teuthology.orchestra.console:Power on smithi031
2023-07-15T02:30:26.320 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-15T02:30:26.346 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-15T02:30:30.131 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-15T02:30:30.343 INFO:teuthology.orchestra.console:Power off for smithi037 completed
2023-07-15T02:30:30.370 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-15T02:30:30.445 INFO:teuthology.orchestra.console:Power on smithi037
2023-07-15T02:30:30.446 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-07-15T02:30:30.469 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-15T02:30:34.484 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-15T02:30:34.494 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-15T02:30:34.696 INFO:teuthology.orchestra.console:Power on for smithi031 completed
2023-07-15T02:30:34.704 INFO:teuthology.orchestra.console:Power on for smithi037 completed
2023-07-15T02:30:34.798 INFO:teuthology.provision.fog.smithi031:Waiting for deploy to finish
2023-07-15T02:30:34.798 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-07-15T02:30:34.805 INFO:teuthology.provision.fog.smithi037:Waiting for deploy to finish
2023-07-15T02:30:34.805 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /task/active None
2023-07-15T02:30:49.885 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /task/active None
2023-07-15T02:30:49.887 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-07-15T02:31:04.968 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-07-15T02:31:04.970 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /task/active None
2023-07-15T02:31:20.064 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-07-15T02:31:20.066 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /task/active None
2023-07-15T02:31:35.148 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-07-15T02:31:35.149 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /task/active None
2023-07-15T02:31:50.229 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /task/active None
2023-07-15T02:31:50.231 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-07-15T02:32:05.308 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-07-15T02:32:05.309 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /task/active None
2023-07-15T02:32:20.386 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-07-15T02:32:20.387 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /task/active None
2023-07-15T02:32:35.458 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-07-15T02:32:35.459 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /task/active None
2023-07-15T02:32:50.532 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /task/active None
2023-07-15T02:32:50.534 DEBUG:teuthology.provision.fog.smithi031:FOG request: GET /task/active None
2023-07-15T02:32:50.605 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-15T02:32:50.887 ERROR:teuthology.orchestra.connection:Error authenticating with smithi031.front.sepia.ceph.com: Authentication failed.
2023-07-15T02:32:50.889 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-07-15T02:32:50.890 DEBUG:teuthology.orchestra.run.smithi031:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-07-15T02:32:50.891 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-15T02:32:51.896 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-15T02:32:52.901 WARNING:paramiko.transport:Oops, unhandled type 3 ('unimplemented')
2023-07-15T02:33:05.604 DEBUG:teuthology.provision.fog.smithi037:FOG request: GET /task/active None
2023-07-15T02:33:05.673 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-15T02:33:08.756 WARNING:teuthology.provision.fog.smithi037:[Errno None] Unable to connect to port 22 on 172.21.15.37
2023-07-15T02:33:14.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-15T02:33:17.812 WARNING:teuthology.provision.fog.smithi037:[Errno None] Unable to connect to port 22 on 172.21.15.37
2023-07-15T02:33:23.814 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-15T02:33:26.864 WARNING:teuthology.provision.fog.smithi037:[Errno None] Unable to connect to port 22 on 172.21.15.37
2023-07-15T02:33:32.866 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-15T02:33:35.920 WARNING:teuthology.provision.fog.smithi037:[Errno None] Unable to connect to port 22 on 172.21.15.37
2023-07-15T02:33:41.922 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-15T02:34:22.192 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104)
2023-07-15T02:34:22.193 ERROR:teuthology.provision.fog:[Errno 104] Connection reset by peer on smithi031
2023-07-15T02:34:31.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com'
2023-07-15T02:34:31.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-15T02:34:31.526 DEBUG:teuthology.orchestra.run.smithi031:> true
2023-07-15T02:34:31.866 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com'
2023-07-15T02:34:31.866 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-07-15T02:34:31.866 DEBUG:teuthology.orchestra.run.smithi031:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-07-15T02:34:31.885 INFO:teuthology.provision.fog.smithi031:Node is ready
2023-07-15T02:34:31.886 DEBUG:teuthology.orchestra.run.smithi031:> hostname
2023-07-15T02:34:31.945 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031.front.sepia.ceph.com
2023-07-15T02:34:31.946 DEBUG:teuthology.orchestra.run.smithi031:> grep smithi031.front.sepia.ceph.com /etc/hosts
2023-07-15T02:34:32.006 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31 smithi031.front.sepia.ceph.com smithi031
2023-07-15T02:34:32.006 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname smithi031
2023-07-15T02:34:32.148 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i -e 's/smithi031.front.sepia.ceph.com/smithi031/g' /etc/hosts
2023-07-15T02:34:32.181 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i -e 's/172.21.15.31/172.21.15.31/g' /etc/hosts
2023-07-15T02:34:32.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i -e 's/smithi031.front.sepia.ceph.com/smithi031/g' /etc/hostname
2023-07-15T02:34:32.322 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname smithi031
2023-07-15T02:34:32.391 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release
2023-07-15T02:34:32.449 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="Red Hat Enterprise Linux"
2023-07-15T02:34:32.450 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="8.6 (Ootpa)"
2023-07-15T02:34:32.450 INFO:teuthology.orchestra.run.smithi031.stdout:ID="rhel"
2023-07-15T02:34:32.450 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="fedora"
2023-07-15T02:34:32.450 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="8.6"
2023-07-15T02:34:32.451 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el8"
2023-07-15T02:34:32.451 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-07-15T02:34:32.451 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31"
2023-07-15T02:34:32.451 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-07-15T02:34:32.451 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://www.redhat.com/"
2023-07-15T02:34:32.451 INFO:teuthology.orchestra.run.smithi031.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-07-15T02:34:32.451 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-07-15T02:34:32.452 INFO:teuthology.orchestra.run.smithi031.stdout:
2023-07-15T02:34:32.452 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-07-15T02:34:32.452 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-07-15T02:34:32.452 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-07-15T02:34:32.452 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-07-15T02:34:32.453 INFO:teuthology.provision.fog.smithi031:Deploy complete!
2023-07-15T02:34:32.454 DEBUG:teuthology.parallel:result is None
2023-07-15T02:34:41.923 WARNING:teuthology.provision.fog.smithi037:timed out
2023-07-15T02:34:47.925 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-15T02:34:48.268 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-07-15T02:34:48.269 DEBUG:teuthology.orchestra.run.smithi037:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-07-15T02:34:48.575 INFO:teuthology.provision.fog.smithi037:Node is ready
2023-07-15T02:34:48.576 DEBUG:teuthology.orchestra.run.smithi037:> hostname
2023-07-15T02:34:48.596 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037.front.sepia.ceph.com
2023-07-15T02:34:48.596 DEBUG:teuthology.orchestra.run.smithi037:> grep smithi037.front.sepia.ceph.com /etc/hosts
2023-07-15T02:34:48.654 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37 smithi037.front.sepia.ceph.com smithi037
2023-07-15T02:34:48.654 DEBUG:teuthology.orchestra.run.smithi037:> sudo hostname smithi037
2023-07-15T02:34:48.920 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i -e 's/smithi037.front.sepia.ceph.com/smithi037/g' /etc/hosts
2023-07-15T02:34:48.951 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i -e 's/172.21.15.37/172.21.15.37/g' /etc/hosts
2023-07-15T02:34:49.022 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i -e 's/smithi037.front.sepia.ceph.com/smithi037/g' /etc/hostname
2023-07-15T02:34:49.092 DEBUG:teuthology.orchestra.run.smithi037:> sudo hostname smithi037
2023-07-15T02:34:49.160 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release
2023-07-15T02:34:49.221 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="Red Hat Enterprise Linux"
2023-07-15T02:34:49.221 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="8.6 (Ootpa)"
2023-07-15T02:34:49.221 INFO:teuthology.orchestra.run.smithi037.stdout:ID="rhel"
2023-07-15T02:34:49.221 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="fedora"
2023-07-15T02:34:49.222 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="8.6"
2023-07-15T02:34:49.222 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el8"
2023-07-15T02:34:49.222 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-07-15T02:34:49.222 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31"
2023-07-15T02:34:49.222 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-07-15T02:34:49.223 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://www.redhat.com/"
2023-07-15T02:34:49.223 INFO:teuthology.orchestra.run.smithi037.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-07-15T02:34:49.223 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-07-15T02:34:49.223 INFO:teuthology.orchestra.run.smithi037.stdout:
2023-07-15T02:34:49.224 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-07-15T02:34:49.224 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-07-15T02:34:49.224 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-07-15T02:34:49.225 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-07-15T02:34:49.225 INFO:teuthology.provision.fog.smithi037:Deploy complete!
2023-07-15T02:34:49.226 DEBUG:teuthology.parallel:result is None
2023-07-15T02:34:49.384 INFO:teuthology.lock.ops:Checking smithi031.front.sepia.ceph.com
2023-07-15T02:34:49.384 INFO:teuthology.lock.ops:New key found. Updating...
2023-07-15T02:34:49.412 INFO:teuthology.lock.ops:Checking smithi037.front.sepia.ceph.com
2023-07-15T02:34:49.412 INFO:teuthology.lock.ops:New key found. Updating...
2023-07-15T02:34:49.429 INFO:teuthology.lock.ops:Updating [smithi031.front.sepia.ceph.com]: set os type and version on server
2023-07-15T02:34:49.430 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com'
2023-07-15T02:34:49.431 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-15T02:34:49.497 DEBUG:teuthology.orchestra.run.smithi031:> true
2023-07-15T02:34:49.572 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com'
2023-07-15T02:34:49.573 DEBUG:teuthology.orchestra.run.smithi031:> uname -m
2023-07-15T02:34:49.632 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64
2023-07-15T02:34:49.633 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release
2023-07-15T02:34:49.692 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="Red Hat Enterprise Linux"
2023-07-15T02:34:49.693 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="8.6 (Ootpa)"
2023-07-15T02:34:49.693 INFO:teuthology.orchestra.run.smithi031.stdout:ID="rhel"
2023-07-15T02:34:49.693 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="fedora"
2023-07-15T02:34:49.693 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="8.6"
2023-07-15T02:34:49.693 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el8"
2023-07-15T02:34:49.694 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-07-15T02:34:49.694 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31"
2023-07-15T02:34:49.694 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-07-15T02:34:49.694 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://www.redhat.com/"
2023-07-15T02:34:49.694 INFO:teuthology.orchestra.run.smithi031.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-07-15T02:34:49.694 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-07-15T02:34:49.694 INFO:teuthology.orchestra.run.smithi031.stdout:
2023-07-15T02:34:49.695 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-07-15T02:34:49.695 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-07-15T02:34:49.695 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-07-15T02:34:49.695 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-07-15T02:34:49.695 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server
2023-07-15T02:34:49.720 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: set os type and version on server
2023-07-15T02:34:49.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-07-15T02:34:49.721 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-07-15T02:34:49.787 DEBUG:teuthology.orchestra.run.smithi037:> true
2023-07-15T02:34:49.860 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-07-15T02:34:49.861 DEBUG:teuthology.orchestra.run.smithi037:> uname -m
2023-07-15T02:34:49.920 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64
2023-07-15T02:34:49.920 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release
2023-07-15T02:34:49.979 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="Red Hat Enterprise Linux"
2023-07-15T02:34:49.979 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="8.6 (Ootpa)"
2023-07-15T02:34:49.980 INFO:teuthology.orchestra.run.smithi037.stdout:ID="rhel"
2023-07-15T02:34:49.980 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="fedora"
2023-07-15T02:34:49.980 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="8.6"
2023-07-15T02:34:49.980 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el8"
2023-07-15T02:34:49.980 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-07-15T02:34:49.980 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31"
2023-07-15T02:34:49.980 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-07-15T02:34:49.981 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://www.redhat.com/"
2023-07-15T02:34:49.981 INFO:teuthology.orchestra.run.smithi037.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-07-15T02:34:49.981 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-07-15T02:34:49.981 INFO:teuthology.orchestra.run.smithi037.stdout:
2023-07-15T02:34:49.981 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-07-15T02:34:49.981 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-07-15T02:34:49.981 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-07-15T02:34:49.981 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-07-15T02:34:49.982 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server
2023-07-15T02:34:50.003 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-15T02:34:50.004 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-15T02:34:50.110 INFO:teuthology.dispatcher.supervisor:Running job 7338301
2023-07-15T02:34:50.110 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-07-14_23:37:57-fs-wip-yuri8-testing-2023-07-14-0803-reef-distro-default-smithi/7338301 --name yuriw-2023-07-14_23:37:57-fs-wip-yuri8-testing-2023-07-14-0803-reef-distro-default-smithi --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} -- /home/teuthworker/archive/yuriw-2023-07-14_23:37:57-fs-wip-yuri8-testing-2023-07-14-0803-reef-distro-default-smithi/7338301/orig.config.yaml
2023-07-15T02:34:50.114 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-07-14_23:37:57-fs-wip-yuri8-testing-2023-07-14-0803-reef-distro-default-smithi/7338301
2023-07-15T02:34:50.115 INFO:teuthology.dispatcher.supervisor:Job PID: 30501
2023-07-15T02:34:50.115 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-07-15T02:36:50.215 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-15T02:36:50.217 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-15T02:38:50.401 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-15T02:38:50.402 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-15T02:40:50.564 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-15T02:40:50.565 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-15T02:42:50.733 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-15T02:42:50.734 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-15T02:44:50.908 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-15T02:44:50.938 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-15T02:46:51.125 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-15T02:46:51.125 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-15T02:48:51.301 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-15T02:48:51.302 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-15T02:50:51.473 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-15T02:50:51.474 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-15T02:52:51.645 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-07-15T02:52:51.645 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None)
2023-07-15T02:52:51.713 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120
2023-07-15T02:52:51.888 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-07-15T02:52:51.888 INFO:teuthology.nuke:Checking targets against current locks
2023-07-15T02:52:51.945 DEBUG:teuthology.nuke:shortname: smithi031
2023-07-15T02:52:51.946 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-15T02:52:51.966 DEBUG:teuthology.nuke:shortname: smithi037
2023-07-15T02:52:51.966 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-07-15T02:52:51.969 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-14_23:37:57-fs-wip-yuri8-testing-2023-07-14-0803-reef-distro-default-smithi/7338301', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-15 02:30:16.450085', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}
2023-07-15T02:52:51.987 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-14_23:37:57-fs-wip-yuri8-testing-2023-07-14-0803-reef-distro-default-smithi/7338301', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-15 02:30:16.447697', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='}
2023-07-15T02:52:52.017 INFO:teuthology.orchestra.console:Power off smithi031
2023-07-15T02:52:52.017 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-15T02:52:52.039 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-15T02:52:52.057 INFO:teuthology.orchestra.console:Power off smithi037
2023-07-15T02:52:52.058 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-07-15T02:52:52.078 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-07-15T02:52:52.168 INFO:teuthology.orchestra.console:Power off for smithi031 completed
2023-07-15T02:52:52.199 INFO:teuthology.orchestra.console:Power off for smithi037 completed
2023-07-15T02:52:52.329 INFO:teuthology.lock.ops:unlocked: smithi031.front.sepia.ceph.com
2023-07-15T02:52:52.351 INFO:teuthology.lock.ops:unlocked: smithi037.front.sepia.ceph.com