2023-08-26T17:46:47.797 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-26T17:46:47.798 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-26T17:46:47.905 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc38520b1f0>, signals=[15]) 2023-08-26T17:46:47.907 INFO:teuthology.lock.ops:Start node 'smithi107.front.sepia.ceph.com' reimaging 2023-08-26T17:46:47.907 INFO:teuthology.lock.ops:Updating [smithi107.front.sepia.ceph.com]: reset os type and version on server 2023-08-26T17:46:47.907 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2023-08-26T17:46:47.929 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /host {"name": "smithi107"} 2023-08-26T17:46:47.998 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /image {"name": "smithi_ubuntu_22.04"} 2023-08-26T17:46:48.039 DEBUG:teuthology.provision.fog.smithi107:Requesting image smithi_ubuntu_22.04 (ID 6) 2023-08-26T17:46:48.039 DEBUG:teuthology.provision.fog.smithi107:FOG request: PUT /host/212 {"imageID": 6} 2023-08-26T17:46:48.080 INFO:teuthology.provision.fog.smithi107:Scheduling deploy of ubuntu 22.04 2023-08-26T17:46:48.081 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None 2023-08-26T17:46:48.130 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /tasktype {"name": "deploy"} 2023-08-26T17:46:48.168 DEBUG:teuthology.provision.fog.smithi107:FOG request: POST /host/212/task {"taskTypeID": 1} 2023-08-26T17:46:48.213 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None 2023-08-26T17:46:48.291 INFO:teuthology.orchestra.console.smithi107:Power off 2023-08-26T17:46:48.292 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-26T17:46:48.313 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2023-08-26T17:46:48.313 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-26T17:46:48.340 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2023-08-26T17:46:48.442 INFO:teuthology.orchestra.console.smithi107:Power off completed 2023-08-26T17:46:48.543 INFO:teuthology.orchestra.console.smithi107:Power on 2023-08-26T17:46:48.543 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2023-08-26T17:46:48.568 DEBUG:teuthology.orchestra.console.smithi107:power on output: Chassis Power Control: Up/On 2023-08-26T17:46:48.569 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-26T17:46:48.590 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2023-08-26T17:46:52.592 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-26T17:46:52.701 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2023-08-26T17:46:56.706 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-26T17:46:56.814 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2023-08-26T17:46:56.915 INFO:teuthology.orchestra.console.smithi107:Power on completed 2023-08-26T17:46:57.015 INFO:teuthology.provision.fog.smithi107:Waiting for deploy to finish 2023-08-26T17:46:57.016 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None 2023-08-26T17:47:12.080 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None 2023-08-26T17:47:27.144 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None 2023-08-26T17:47:42.205 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None 2023-08-26T17:47:57.272 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None 2023-08-26T17:48:12.339 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None 2023-08-26T17:48:27.413 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None 2023-08-26T17:48:42.480 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None 2023-08-26T17:48:57.548 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None 2023-08-26T17:49:12.613 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None 2023-08-26T17:49:27.680 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None 2023-08-26T17:49:42.744 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None 2023-08-26T17:49:57.809 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None 2023-08-26T17:50:12.880 DEBUG:teuthology.provision.fog.smithi107:FOG request: GET /task/active None 2023-08-26T17:50:12.942 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-26T17:50:16.016 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-08-26T17:50:22.020 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-26T17:50:25.072 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-08-26T17:50:31.076 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-26T17:50:34.128 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-08-26T17:50:40.132 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-26T17:50:43.184 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-08-26T17:50:49.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-26T17:50:52.240 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-08-26T17:50:58.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-26T17:51:01.296 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-08-26T17:51:07.300 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-26T17:51:10.352 WARNING:teuthology.provision.fog.smithi107:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-08-26T17:51:16.357 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-26T17:52:16.417 WARNING:teuthology.provision.fog.smithi107:timed out 2023-08-26T17:52:22.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-26T17:52:22.703 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-08-26T17:52:22.704 DEBUG:teuthology.orchestra.run.smithi107:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2023-08-26T17:52:23.155 INFO:teuthology.provision.fog.smithi107:Node is ready 2023-08-26T17:52:23.156 DEBUG:teuthology.orchestra.run.smithi107:> hostname 2023-08-26T17:52:23.207 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107.front.sepia.ceph.com 2023-08-26T17:52:23.207 DEBUG:teuthology.orchestra.run.smithi107:> grep smithi107.front.sepia.ceph.com /etc/hosts 2023-08-26T17:52:23.255 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107 smithi107.front.sepia.ceph.com smithi107 2023-08-26T17:52:23.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname smithi107 2023-08-26T17:52:23.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i -e 's/smithi107.front.sepia.ceph.com/smithi107/g' /etc/hosts 2023-08-26T17:52:23.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i -e 's/172.21.15.107/172.21.15.107/g' /etc/hosts 2023-08-26T17:52:23.475 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i -e 's/smithi107.front.sepia.ceph.com/smithi107/g' /etc/hostname 2023-08-26T17:52:23.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname smithi107 2023-08-26T17:52:23.586 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2023-08-26T17:52:23.640 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-08-26T17:52:23.640 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="Ubuntu" 2023-08-26T17:52:23.640 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="22.04" 2023-08-26T17:52:23.640 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-08-26T17:52:23.641 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_CODENAME=jammy 2023-08-26T17:52:23.641 INFO:teuthology.orchestra.run.smithi107.stdout:ID=ubuntu 2023-08-26T17:52:23.641 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE=debian 2023-08-26T17:52:23.641 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://www.ubuntu.com/" 2023-08-26T17:52:23.641 INFO:teuthology.orchestra.run.smithi107.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-08-26T17:52:23.642 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-08-26T17:52:23.642 INFO:teuthology.orchestra.run.smithi107.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-08-26T17:52:23.642 INFO:teuthology.orchestra.run.smithi107.stdout:UBUNTU_CODENAME=jammy 2023-08-26T17:52:23.643 INFO:teuthology.provision.fog.smithi107:Deploy complete! 2023-08-26T17:52:23.643 DEBUG:teuthology.parallel:result is None 2023-08-26T17:52:23.768 INFO:teuthology.lock.ops:Checking smithi107.front.sepia.ceph.com 2023-08-26T17:52:23.768 INFO:teuthology.lock.ops:New key found. Updating... 2023-08-26T17:52:23.790 INFO:teuthology.lock.ops:Updating [smithi107.front.sepia.ceph.com]: set os type and version on server 2023-08-26T17:52:23.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-08-26T17:52:23.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-26T17:52:23.868 DEBUG:teuthology.orchestra.run.smithi107:> true 2023-08-26T17:52:23.955 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-08-26T17:52:23.956 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2023-08-26T17:52:24.003 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2023-08-26T17:52:24.003 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2023-08-26T17:52:24.051 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-08-26T17:52:24.051 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="Ubuntu" 2023-08-26T17:52:24.051 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="22.04" 2023-08-26T17:52:24.051 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-08-26T17:52:24.052 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_CODENAME=jammy 2023-08-26T17:52:24.052 INFO:teuthology.orchestra.run.smithi107.stdout:ID=ubuntu 2023-08-26T17:52:24.052 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE=debian 2023-08-26T17:52:24.052 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://www.ubuntu.com/" 2023-08-26T17:52:24.053 INFO:teuthology.orchestra.run.smithi107.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-08-26T17:52:24.053 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-08-26T17:52:24.053 INFO:teuthology.orchestra.run.smithi107.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-08-26T17:52:24.053 INFO:teuthology.orchestra.run.smithi107.stdout:UBUNTU_CODENAME=jammy 2023-08-26T17:52:24.054 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2023-08-26T17:52:24.073 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-26T17:52:24.156 INFO:teuthology.dispatcher.supervisor:Running job 7380814 2023-08-26T17:52:24.157 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-08-25_18:09:36-rados-wip-yuri-testing-2023-08-25-0809-distro-default-smithi/7380814 --name yuriw-2023-08-25_18:09:36-rados-wip-yuri-testing-2023-08-25-0809-distro-default-smithi --description rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} -- /home/teuthworker/archive/yuriw-2023-08-25_18:09:36-rados-wip-yuri-testing-2023-08-25-0809-distro-default-smithi/7380814/orig.config.yaml 2023-08-26T17:52:24.159 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-08-25_18:09:36-rados-wip-yuri-testing-2023-08-25-0809-distro-default-smithi/7380814 2023-08-26T17:52:24.160 INFO:teuthology.dispatcher.supervisor:Job PID: 2417232 2023-08-26T17:52:24.160 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-08-26T17:54:24.254 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-26T17:54:24.317 ERROR:teuthology.dispatcher.supervisor:Child exited with code 120 2023-08-26T17:54:24.376 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2023-08-26T17:54:24.376 INFO:teuthology.nuke:Checking targets against current locks 2023-08-26T17:54:24.424 DEBUG:teuthology.nuke:shortname: smithi107 2023-08-26T17:54:24.425 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-26T17:54:24.442 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-25_18:09:36-rados-wip-yuri-testing-2023-08-25-0809-distro-default-smithi/7380814', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-26 17:46:47.037060', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOjAhT0myoOjkUb4sm1Z78yHPzz7si2HS8VdnF14B+uq'} 2023-08-26T17:54:24.487 INFO:teuthology.orchestra.console.smithi107:Power off 2023-08-26T17:54:24.487 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-26T17:54:24.509 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2023-08-26T17:54:24.510 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-26T17:54:24.528 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2023-08-26T17:54:24.629 INFO:teuthology.orchestra.console.smithi107:Power off completed 2023-08-26T17:54:24.775 INFO:teuthology.lock.ops:unlocked: smithi107.front.sepia.ceph.com