2023-07-17T23:16:14.221 INFO:root:teuthology version: 0.0.1.dev147+gff9991b 2023-07-17T23:16:14.222 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-17T23:16:14.223 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-17T23:16:24.396 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff0bace38b0>, signals=[15]) 2023-07-17T23:16:24.399 INFO:teuthology.lock.ops:Start node 'smithi061.front.sepia.ceph.com' reimaging 2023-07-17T23:16:24.399 INFO:teuthology.lock.ops:Updating [smithi061.front.sepia.ceph.com]: reset os type and version on server 2023-07-17T23:16:24.399 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server 2023-07-17T23:16:34.433 INFO:teuthology.lock.ops:Start node 'smithi099.front.sepia.ceph.com' reimaging 2023-07-17T23:16:34.433 INFO:teuthology.lock.ops:Updating [smithi099.front.sepia.ceph.com]: reset os type and version on server 2023-07-17T23:16:34.434 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2023-07-17T23:16:34.435 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /host {"name": "smithi061"} 2023-07-17T23:16:34.465 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /host {"name": "smithi099"} 2023-07-17T23:16:34.515 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /image {"name": "smithi_rhel_8.6"} 2023-07-17T23:16:34.527 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /image {"name": "smithi_rhel_8.6"} 2023-07-17T23:16:34.562 DEBUG:teuthology.provision.fog.smithi061:Requesting image smithi_rhel_8.6 (ID 5) 2023-07-17T23:16:34.562 DEBUG:teuthology.provision.fog.smithi061:FOG request: PUT /host/166 {"imageID": 5} 2023-07-17T23:16:34.566 DEBUG:teuthology.provision.fog.smithi099:Requesting image smithi_rhel_8.6 (ID 5) 2023-07-17T23:16:34.566 DEBUG:teuthology.provision.fog.smithi099:FOG request: PUT /host/204 {"imageID": 5} 2023-07-17T23:16:34.599 INFO:teuthology.provision.fog.smithi061:Scheduling deploy of rhel 8.6 2023-07-17T23:16:34.600 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None 2023-07-17T23:16:34.616 INFO:teuthology.provision.fog.smithi099:Scheduling deploy of rhel 8.6 2023-07-17T23:16:34.617 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /task/active None 2023-07-17T23:16:34.663 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /tasktype {"name": "deploy"} 2023-07-17T23:16:34.675 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /tasktype {"name": "deploy"} 2023-07-17T23:16:34.699 DEBUG:teuthology.provision.fog.smithi061:FOG request: POST /host/166/task {"taskTypeID": 1} 2023-07-17T23:16:34.715 DEBUG:teuthology.provision.fog.smithi099:FOG request: POST /host/204/task {"taskTypeID": 1} 2023-07-17T23:16:34.759 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None 2023-07-17T23:16:34.766 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /task/active None 2023-07-17T23:16:34.881 INFO:teuthology.orchestra.console:Power off smithi061 2023-07-17T23:16:34.882 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-17T23:16:34.898 INFO:teuthology.orchestra.console:Power off smithi099 2023-07-17T23:16:34.899 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-17T23:16:34.907 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:16:34.923 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:16:38.940 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:16:38.951 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:16:43.058 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:16:43.067 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:16:48.170 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:16:48.181 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:16:53.284 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:16:53.294 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:16:53.506 INFO:teuthology.orchestra.console:Power off for smithi099 completed 2023-07-17T23:16:53.607 INFO:teuthology.orchestra.console:Power on smithi099 2023-07-17T23:16:53.607 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-07-17T23:16:53.629 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:16:57.658 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:16:58.397 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:17:01.772 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:17:03.509 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:17:06.884 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:17:08.623 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:17:11.996 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:17:12.207 INFO:teuthology.orchestra.console:Power on for smithi099 completed 2023-07-17T23:17:12.309 INFO:teuthology.provision.fog.smithi099:Waiting for deploy to finish 2023-07-17T23:17:12.309 DEBUG:teuthology.provision.fog.smithi099:FOG request: GET /task/active None 2023-07-17T23:17:17.317 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:17:17.372 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-17T23:17:20.432 WARNING:teuthology.provision.fog.smithi099:[Errno None] Unable to connect to port 22 on 172.21.15.99 2023-07-17T23:17:22.431 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:17:26.437 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-17T23:17:27.543 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:17:29.488 WARNING:teuthology.provision.fog.smithi099:[Errno None] Unable to connect to port 22 on 172.21.15.99 2023-07-17T23:17:32.656 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:17:35.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-17T23:17:37.767 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:17:38.544 WARNING:teuthology.provision.fog.smithi099:[Errno None] Unable to connect to port 22 on 172.21.15.99 2023-07-17T23:17:42.879 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:17:44.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-17T23:17:49.554 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:17:50.664 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-07-17T23:17:50.665 ERROR:teuthology.orchestra.console:Failed to power off smithi061 2023-07-17T23:17:50.665 INFO:teuthology.orchestra.console:Power on smithi061 2023-07-17T23:17:50.666 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-07-17T23:17:52.626 WARNING:teuthology.provision.fog.smithi099:[Errno None] Unable to connect to port 22 on 172.21.15.99 2023-07-17T23:17:52.630 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:17:56.656 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:17:58.629 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-17T23:18:01.680 WARNING:teuthology.provision.fog.smithi099:[Errno None] Unable to connect to port 22 on 172.21.15.99 2023-07-17T23:18:01.768 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:18:06.880 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:18:07.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-17T23:18:11.992 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:18:17.103 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:18:17.314 INFO:teuthology.orchestra.console:Power on for smithi061 completed 2023-07-17T23:18:17.415 INFO:teuthology.provision.fog.smithi061:Waiting for deploy to finish 2023-07-17T23:18:17.416 DEBUG:teuthology.provision.fog.smithi061:FOG request: GET /task/active None 2023-07-17T23:18:27.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-17T23:18:35.572 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-07-17T23:18:41.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-17T23:18:49.648 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-07-17T23:18:55.652 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-17T23:19:03.729 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-07-17T23:19:07.688 WARNING:teuthology.provision.fog.smithi099:timed out 2023-07-17T23:19:09.732 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-17T23:19:13.692 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-17T23:19:19.121 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-07-17T23:19:19.122 DEBUG:teuthology.orchestra.run.smithi099:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-07-17T23:19:19.368 INFO:teuthology.provision.fog.smithi099:Node is ready 2023-07-17T23:19:19.368 DEBUG:teuthology.orchestra.run.smithi099:> hostname 2023-07-17T23:19:19.387 INFO:teuthology.orchestra.run.smithi099.stdout:smithi099.front.sepia.ceph.com 2023-07-17T23:19:19.387 DEBUG:teuthology.orchestra.run.smithi099:> grep smithi099.front.sepia.ceph.com /etc/hosts 2023-07-17T23:19:19.446 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.15.99 smithi099.front.sepia.ceph.com smithi099 2023-07-17T23:19:19.446 DEBUG:teuthology.orchestra.run.smithi099:> sudo hostname smithi099 2023-07-17T23:19:19.547 DEBUG:teuthology.orchestra.run.smithi099:> sudo sed -i -e 's/smithi099.front.sepia.ceph.com/smithi099/g' /etc/hosts 2023-07-17T23:19:19.577 DEBUG:teuthology.orchestra.run.smithi099:> sudo sed -i -e 's/172.21.15.99/172.21.15.99/g' /etc/hosts 2023-07-17T23:19:19.653 DEBUG:teuthology.orchestra.run.smithi099:> sudo sed -i -e 's/smithi099.front.sepia.ceph.com/smithi099/g' /etc/hostname 2023-07-17T23:19:19.721 DEBUG:teuthology.orchestra.run.smithi099:> sudo hostname smithi099 2023-07-17T23:19:19.788 DEBUG:teuthology.orchestra.run.smithi099:> cat /etc/os-release 2023-07-17T23:19:19.846 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="CentOS Stream" 2023-07-17T23:19:19.846 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="8" 2023-07-17T23:19:19.846 INFO:teuthology.orchestra.run.smithi099.stdout:ID="centos" 2023-07-17T23:19:19.846 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE="rhel fedora" 2023-07-17T23:19:19.846 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="8" 2023-07-17T23:19:19.847 INFO:teuthology.orchestra.run.smithi099.stdout:PLATFORM_ID="platform:el8" 2023-07-17T23:19:19.847 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="CentOS Stream 8" 2023-07-17T23:19:19.847 INFO:teuthology.orchestra.run.smithi099.stdout:ANSI_COLOR="0;31" 2023-07-17T23:19:19.847 INFO:teuthology.orchestra.run.smithi099.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-07-17T23:19:19.847 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://centos.org/" 2023-07-17T23:19:19.847 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-17T23:19:19.847 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-17T23:19:19.848 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-07-17T23:19:19.850 ERROR:teuthology.dispatcher.supervisor:Reimaging error. Nuking machines... Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/fog.py", line 92, in create self._verify_installed_os() File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/fog.py", line 348, in _verify_installed_os raise RuntimeError( RuntimeError: Expected smithi099's OS to be rhel 8.6 but found centos 8 2023-07-17T23:19:19.850 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-17T23:19:19.851 DEBUG:urllib3.util.retry:Converted retries value: 10 -> Retry(total=10, connect=None, read=None, redirect=None, status=None) 2023-07-17T23:19:24.945 INFO:teuthology.nuke:Checking targets against current locks 2023-07-17T23:19:45.022 DEBUG:teuthology.nuke:shortname: smithi061 2023-07-17T23:19:45.022 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-17T23:19:45.058 DEBUG:teuthology.nuke:shortname: smithi099 2023-07-17T23:19:45.059 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-17T23:19:45.065 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi061.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-17_18:51:04-rados-wip-yuri6-testing-2023-07-17-0838-reef-distro-default-smithi/7342207', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-17 23:16:13.404493', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyQA01sOb7bmWSWbLI4J9aUHL7laJ25xs+Brwqle+GwZsEHQIIZIZ6N37b3Cdg5Qtw+Oo5tRgyQ/wQpJ4maXv8g1Zh7fHyAUq848Wj0VcdlrlK2ofBIVSI1cLU1ibFdshtYB9AbDitsB8RPOoHqCVkWjVSbpMrMeuXD+ZDlJpakwjwPd/I+XO1Yxa1mDpzXvUOa/ObGayz5BB2s2quGgfe1Gpv78hFbOKPDdwU79a48gKOt20+zKDvoRCQUvoLFHWvQov/FU1ncNnFeG9SIL4q8QeyQqXv857EjOlYmNsbR8raxEqHWi/Ct/JG6TjiLgLrxi//FpuwM7Lx6m4SHO9n6eqrb/F1HRcHOa49fBo6v092ZBJ+k5y44EuktVqFGphYvJUHiFpFlWEB9V4bgW+VRfcolIRmOm6/qzTeOC9XBYxDroQOxDNfcDxp1aMkY8gLK3ZajnZxSjWtMJR+nYk5ft0Z9GrA+nTYTMZ8okEaUuU023Qt7uisxTT+A9v30Q0='} 2023-07-17T23:19:45.087 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-17_18:51:04-rados-wip-yuri6-testing-2023-07-17-0838-reef-distro-default-smithi/7342207', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': '', 'os_version': '', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-17 23:16:13.406256', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3lnD1iV984CAeIq+8QdlJuqhAMzkrV2ezN3hISIaA9jUZAPkBA7dLBHYkKTMC93w0cbKGcTm2g4f3sLI/WUoKjGJtMfUE1mj06gPRsPslFKzDnpw2V5MHBaTKTbAE56ed4nHwswmmmHnXUWbgQwarfWIRGioj8CUa20H9+QtY87fiVpFpfjHDsQn3rYWNZjgeu0RvuoAZMjQ987fEhbA1/RWs053Fc7YhCbSFxJTD04ZgIVtNlYx0NFup/KvDX3qKmdvvKCLmNmCi8ppSRvojtzo+hshtF1UbWY+OEyvUkUzHQ/c5aZru2JkRExcIpfaJkmnoboecEKMFulbofzVTuz/btQhOG4+1w+6f7wwbg/0HTwhA6jY749crdSwGqpWBnVtRAXlrImGmsB13ASYPiiydguGAw2MKL7WwBZaantwgSEInBkhRkwL4hzkrv8gF140vLR0EE7uWbX+1aI9QEJAuGjib9dFqciAQ5mw73vKsDzBv4oynzJnDrC3Wuts='} 2023-07-17T23:19:45.122 INFO:teuthology.orchestra.console:Power off smithi061 2023-07-17T23:19:45.123 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-17T23:19:45.141 INFO:teuthology.orchestra.console:Power off smithi099 2023-07-17T23:19:45.142 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-17T23:19:45.154 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:19:45.166 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:19:49.181 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:19:49.190 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:19:54.293 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:19:54.303 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:19:59.407 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:19:59.416 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:20:04.519 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:20:04.529 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:20:04.732 INFO:teuthology.orchestra.console:Power off for smithi061 completed 2023-07-17T23:20:09.842 WARNING:teuthology.provision.fog.smithi061:timed out 2023-07-17T23:20:14.877 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:20:14.911 INFO:teuthology.lock.ops:unlocked: smithi061.front.sepia.ceph.com 2023-07-17T23:20:15.845 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-17T23:20:19.989 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:20:25.099 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:20:30.210 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:20:34.160 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-07-17T23:20:35.325 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:20:40.165 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-17T23:20:40.436 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:20:43.216 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-07-17T23:20:45.547 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:20:49.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-17T23:20:54.230 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:20:57.296 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-07-17T23:20:59.344 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:21:03.301 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-17T23:21:08.305 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-17T23:21:09.416 WARNING:teuthology.contextutil:'wait for power off' reached maximum tries (16) after waiting for 60.0 seconds 2023-07-17T23:21:09.417 ERROR:teuthology.orchestra.console:Failed to power off smithi099 2023-07-17T23:21:14.425 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-07-17T23:21:19.486 INFO:teuthology.lock.ops:unlocked: smithi099.front.sepia.ceph.com 2023-07-17T23:21:24.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-17T23:21:27.769 WARNING:teuthology.provision.fog.smithi061:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-07-17T23:21:27.905 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/dispatcher/__init__.py", line 69, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/dispatcher/supervisor.py", line 57, in main reimage(job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/dispatcher/supervisor.py", line 227, in reimage reimaged = lock_ops.reimage_machines(ctx, targets, job_config['machine_type']) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/lock/ops.py", line 325, in reimage_machines reimaged[machine] = machines[machine] File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/__init__.py", line 44, in reimage result = obj.create() File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/fog.py", line 92, in create self._verify_installed_os() File "/home/teuthworker/src/git.ceph.com_teuthology_ff9991b7ed74eda9de4a5215d6d7d2a72f7b8d2b/teuthology/provision/fog.py", line 348, in _verify_installed_os raise RuntimeError( RuntimeError: Expected smithi099's OS to be rhel 8.6 but found centos 8