2023-10-02T17:29:24.929 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-02T17:29:24.930 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-02T17:29:25.050 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb1e01cf670>, signals=[15]) 2023-10-02T17:29:25.050 INFO:teuthology.lock.ops:Start node 'smithi006.front.sepia.ceph.com' reimaging 2023-10-02T17:29:25.050 INFO:teuthology.lock.ops:Updating [smithi006.front.sepia.ceph.com]: reset os type and version on server 2023-10-02T17:29:25.051 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2023-10-02T17:29:25.072 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /host {"name": "smithi006"} 2023-10-02T17:29:25.139 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /image {"name": "smithi_centos_8.stream"} 2023-10-02T17:29:25.177 DEBUG:teuthology.provision.fog.smithi006:Requesting image smithi_centos_8.stream (ID 3) 2023-10-02T17:29:25.178 DEBUG:teuthology.provision.fog.smithi006:FOG request: PUT /host/112 {"imageID": 3} 2023-10-02T17:29:25.236 INFO:teuthology.provision.fog.smithi006:Scheduling deploy of centos 8.stream 2023-10-02T17:29:25.237 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:29:25.354 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /tasktype {"name": "deploy"} 2023-10-02T17:29:25.387 DEBUG:teuthology.provision.fog.smithi006:FOG request: POST /host/112/task {"taskTypeID": 1} 2023-10-02T17:29:25.443 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:29:25.624 INFO:teuthology.orchestra.console.smithi006:Power off 2023-10-02T17:29:25.624 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-02T17:29:25.645 DEBUG:teuthology.orchestra.console.smithi006:power off output: Chassis Power Control: Down/Off 2023-10-02T17:29:25.645 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-02T17:29:25.666 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2023-10-02T17:29:29.666 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-02T17:29:29.774 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2023-10-02T17:29:33.778 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-02T17:29:33.886 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is off 2023-10-02T17:29:33.987 INFO:teuthology.orchestra.console.smithi006:Power off completed 2023-10-02T17:29:34.087 INFO:teuthology.orchestra.console.smithi006:Power on 2023-10-02T17:29:34.087 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-10-02T17:29:34.107 DEBUG:teuthology.orchestra.console.smithi006:power on output: Chassis Power Control: Up/On 2023-10-02T17:29:34.107 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-02T17:29:34.126 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is off 2023-10-02T17:29:38.130 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-02T17:29:38.237 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is off 2023-10-02T17:29:42.242 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-02T17:29:42.350 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2023-10-02T17:29:42.450 INFO:teuthology.orchestra.console.smithi006:Power on completed 2023-10-02T17:29:42.551 INFO:teuthology.provision.fog.smithi006:Waiting for deploy to finish 2023-10-02T17:29:42.551 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:29:57.759 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:30:12.994 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:30:28.206 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:30:43.430 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:30:58.706 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:31:14.006 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:31:29.354 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:32:01.403 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:32:20.066 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:32:39.678 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:32:55.292 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:33:10.627 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:33:25.848 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:33:41.158 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:33:56.436 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:34:11.666 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:34:26.894 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:34:42.094 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:34:57.282 DEBUG:teuthology.provision.fog.smithi006:FOG request: GET /task/active None 2023-10-02T17:34:57.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-02T17:35:00.506 WARNING:teuthology.provision.fog.smithi006:[Errno None] Unable to connect to port 22 on 172.21.15.6 2023-10-02T17:35:06.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-02T17:35:09.562 WARNING:teuthology.provision.fog.smithi006:[Errno None] Unable to connect to port 22 on 172.21.15.6 2023-10-02T17:35:15.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-02T17:35:18.650 WARNING:teuthology.provision.fog.smithi006:[Errno None] Unable to connect to port 22 on 172.21.15.6 2023-10-02T17:35:24.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-02T17:35:27.707 WARNING:teuthology.provision.fog.smithi006:[Errno None] Unable to connect to port 22 on 172.21.15.6 2023-10-02T17:35:33.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-02T17:35:36.766 WARNING:teuthology.provision.fog.smithi006:[Errno None] Unable to connect to port 22 on 172.21.15.6 2023-10-02T17:35:42.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-02T17:36:42.830 WARNING:teuthology.provision.fog.smithi006:timed out 2023-10-02T17:36:48.835 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-02T17:36:49.154 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-10-02T17:36:49.154 DEBUG:teuthology.orchestra.run.smithi006:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-10-02T17:36:49.400 INFO:teuthology.provision.fog.smithi006:Node is ready 2023-10-02T17:36:49.400 DEBUG:teuthology.orchestra.run.smithi006:> hostname 2023-10-02T17:36:49.419 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006.front.sepia.ceph.com 2023-10-02T17:36:49.420 DEBUG:teuthology.orchestra.run.smithi006:> grep smithi006.front.sepia.ceph.com /etc/hosts 2023-10-02T17:36:49.482 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6 smithi006.front.sepia.ceph.com smithi006 2023-10-02T17:36:49.482 DEBUG:teuthology.orchestra.run.smithi006:> sudo hostname smithi006 2023-10-02T17:36:49.590 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i -e 's/smithi006.front.sepia.ceph.com/smithi006/g' /etc/hosts 2023-10-02T17:36:49.619 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i -e 's/172.21.15.6/172.21.15.6/g' /etc/hosts 2023-10-02T17:36:49.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i -e 's/smithi006.front.sepia.ceph.com/smithi006/g' /etc/hostname 2023-10-02T17:36:49.763 DEBUG:teuthology.orchestra.run.smithi006:> sudo hostname smithi006 2023-10-02T17:36:49.833 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2023-10-02T17:36:49.895 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="CentOS Stream" 2023-10-02T17:36:49.895 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="8" 2023-10-02T17:36:49.896 INFO:teuthology.orchestra.run.smithi006.stdout:ID="centos" 2023-10-02T17:36:49.896 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="rhel fedora" 2023-10-02T17:36:49.896 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="8" 2023-10-02T17:36:49.896 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el8" 2023-10-02T17:36:49.896 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-02T17:36:49.896 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2023-10-02T17:36:49.896 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-02T17:36:49.896 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://centos.org/" 2023-10-02T17:36:49.896 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-02T17:36:49.896 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-02T17:36:49.896 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-02T17:36:49.897 INFO:teuthology.provision.fog.smithi006:Deploy complete! 2023-10-02T17:36:49.898 DEBUG:teuthology.parallel:result is None 2023-10-02T17:36:50.006 INFO:teuthology.lock.ops:Checking smithi006.front.sepia.ceph.com 2023-10-02T17:36:50.007 INFO:teuthology.lock.ops:New key found. Updating... 2023-10-02T17:36:50.035 INFO:teuthology.lock.ops:Updating [smithi006.front.sepia.ceph.com]: set os type and version on server 2023-10-02T17:36:50.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-10-02T17:36:50.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-02T17:36:50.104 DEBUG:teuthology.orchestra.run.smithi006:> true 2023-10-02T17:36:50.172 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-10-02T17:36:50.173 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2023-10-02T17:36:50.234 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2023-10-02T17:36:50.234 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2023-10-02T17:36:50.296 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="CentOS Stream" 2023-10-02T17:36:50.296 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="8" 2023-10-02T17:36:50.296 INFO:teuthology.orchestra.run.smithi006.stdout:ID="centos" 2023-10-02T17:36:50.297 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="rhel fedora" 2023-10-02T17:36:50.297 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="8" 2023-10-02T17:36:50.297 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el8" 2023-10-02T17:36:50.297 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-02T17:36:50.297 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2023-10-02T17:36:50.297 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-02T17:36:50.297 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://centos.org/" 2023-10-02T17:36:50.297 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-02T17:36:50.297 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-02T17:36:50.297 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-02T17:36:50.298 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2023-10-02T17:36:50.317 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-02T17:36:50.400 INFO:teuthology.dispatcher.supervisor:Running job 7408531 2023-10-02T17:36:50.400 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/bin/teuthology -v --owner scheduled_adking@teuthology --archive /home/teuthworker/archive/adking-2023-10-02_17:28:22-orch:cephadm-wip-adk3-testing-2023-10-02-1030-distro-default-smithi/7408531 --name adking-2023-10-02_17:28:22-orch:cephadm-wip-adk3-testing-2023-10-02-1030-distro-default-smithi --description orch:cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_orch_cli} -- /home/teuthworker/archive/adking-2023-10-02_17:28:22-orch:cephadm-wip-adk3-testing-2023-10-02-1030-distro-default-smithi/7408531/orig.config.yaml 2023-10-02T17:36:50.403 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/adking-2023-10-02_17:28:22-orch:cephadm-wip-adk3-testing-2023-10-02-1030-distro-default-smithi/7408531 2023-10-02T17:36:50.403 INFO:teuthology.dispatcher.supervisor:Job PID: 268671 2023-10-02T17:36:50.403 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-10-02T17:38:50.503 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-02T17:40:50.671 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-02T17:42:50.835 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-02T17:44:51.023 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-02T17:46:51.193 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-02T17:48:51.363 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-02T17:50:51.539 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-02T17:52:51.703 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-02T17:54:51.865 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-02T17:56:52.002 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-02T17:58:52.071 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-02T18:00:52.245 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-02T18:02:52.413 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-02T18:02:52.479 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1 2023-10-02T18:02:52.536 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2023-10-02T18:02:52.537 INFO:teuthology.nuke:Checking targets against current locks 2023-10-02T18:02:52.578 DEBUG:teuthology.nuke:shortname: smithi006 2023-10-02T18:02:52.579 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-02T18:02:52.600 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2023-10-02_17:28:22-orch:cephadm-wip-adk3-testing-2023-10-02-1030-distro-default-smithi/7408531', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-02 17:29:24.213915', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGB+SIHAPMqJ8kiVhWnEWO5T0qDFeeT2gUhZS+BLwp9KIXTN1jPVfcXn3ew3fKdeSMDDRhWuWTnIQ7CtmbRQ1Vc='} 2023-10-02T18:02:52.652 INFO:teuthology.orchestra.console.smithi006:Power off 2023-10-02T18:02:52.652 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-02T18:02:52.675 DEBUG:teuthology.orchestra.console.smithi006:power off output: Chassis Power Control: Down/Off 2023-10-02T18:02:52.675 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-02T18:02:52.694 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is off 2023-10-02T18:02:52.795 INFO:teuthology.orchestra.console.smithi006:Power off completed 2023-10-02T18:02:52.954 INFO:teuthology.lock.ops:unlocked: smithi006.front.sepia.ceph.com