2023-11-29T09:42:10.021 INFO:root:teuthology version: 0.0.1.dev212+g735a43c
2023-11-29T09:42:10.021 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T09:42:10.161 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f6d349e3d90>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f6d346ebd30>, signals=[15])
2023-11-29T09:42:10.161 INFO:teuthology.lock.ops:Start node 'smithi032.front.sepia.ceph.com' reimaging
2023-11-29T09:42:10.162 INFO:teuthology.lock.ops:Updating [smithi032.front.sepia.ceph.com]: reset os type and version on server
2023-11-29T09:42:10.162 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server
2023-11-29T09:42:10.184 INFO:teuthology.lock.ops:Start node 'smithi196.front.sepia.ceph.com' reimaging
2023-11-29T09:42:10.184 INFO:teuthology.lock.ops:Updating [smithi196.front.sepia.ceph.com]: reset os type and version on server
2023-11-29T09:42:10.185 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server
2023-11-29T09:42:10.307 DEBUG:teuthology.provision.fog.smithi032:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-11-29T09:42:10.317 DEBUG:teuthology.provision.fog.smithi196:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-11-29T09:42:10.355 INFO:teuthology.provision.fog.smithi032:Scheduling deploy of ubuntu 22.04
2023-11-29T09:42:10.362 INFO:teuthology.provision.fog.smithi196:Scheduling deploy of ubuntu 22.04
2023-11-29T09:42:10.881 INFO:teuthology.orchestra.console.smithi032:Power off
2023-11-29T09:42:10.881 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-29T09:42:10.894 INFO:teuthology.orchestra.console.smithi196:Power off
2023-11-29T09:42:10.894 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-29T09:42:10.905 DEBUG:teuthology.orchestra.console.smithi032:power off output: Chassis Power Control: Down/Off
2023-11-29T09:42:10.905 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-29T09:42:10.914 DEBUG:teuthology.orchestra.console.smithi196:power off output: Chassis Power Control: Down/Off
2023-11-29T09:42:10.914 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-29T09:42:10.927 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on
2023-11-29T09:42:10.935 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on
2023-11-29T09:42:14.932 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-29T09:42:14.939 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-29T09:42:15.041 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on
2023-11-29T09:42:15.047 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on
2023-11-29T09:42:19.046 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-29T09:42:19.053 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-29T09:42:19.155 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off
2023-11-29T09:42:19.161 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on
2023-11-29T09:42:19.256 INFO:teuthology.orchestra.console.smithi032:Power off completed
2023-11-29T09:42:19.357 INFO:teuthology.orchestra.console.smithi032:Power on
2023-11-29T09:42:19.357 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-29T09:42:19.378 DEBUG:teuthology.orchestra.console.smithi032:power on output: Chassis Power Control: Up/On
2023-11-29T09:42:19.379 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-29T09:42:19.399 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off
2023-11-29T09:42:23.165 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-29T09:42:23.274 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is off
2023-11-29T09:42:23.375 INFO:teuthology.orchestra.console.smithi196:Power off completed
2023-11-29T09:42:23.400 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-29T09:42:23.476 INFO:teuthology.orchestra.console.smithi196:Power on
2023-11-29T09:42:23.477 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-29T09:42:23.496 DEBUG:teuthology.orchestra.console.smithi196:power on output: Chassis Power Control: Up/On
2023-11-29T09:42:23.496 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-29T09:42:23.509 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off
2023-11-29T09:42:23.515 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is off
2023-11-29T09:42:27.510 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-29T09:42:27.517 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-29T09:42:27.619 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on
2023-11-29T09:42:27.626 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is off
2023-11-29T09:42:27.721 INFO:teuthology.orchestra.console.smithi032:Power on completed
2023-11-29T09:42:27.821 INFO:teuthology.provision.fog.smithi032:Waiting for deploy to finish
2023-11-29T09:42:31.630 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-29T09:42:31.738 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on
2023-11-29T09:42:31.838 INFO:teuthology.orchestra.console.smithi196:Power on completed
2023-11-29T09:42:31.939 INFO:teuthology.provision.fog.smithi196:Waiting for deploy to finish
2023-11-29T09:52:12.470 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-29T09:52:15.524 WARNING:teuthology.provision.fog.smithi032:[Errno None] Unable to connect to port 22 on 172.21.15.32
2023-11-29T09:52:21.530 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-29T09:52:24.610 WARNING:teuthology.provision.fog.smithi032:[Errno None] Unable to connect to port 22 on 172.21.15.32
2023-11-29T09:52:30.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-29T09:52:33.666 WARNING:teuthology.provision.fog.smithi032:[Errno None] Unable to connect to port 22 on 172.21.15.32
2023-11-29T09:52:39.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-29T09:52:42.722 WARNING:teuthology.provision.fog.smithi032:[Errno None] Unable to connect to port 22 on 172.21.15.32
2023-11-29T09:52:48.726 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-29T09:53:16.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-29T09:53:19.202 WARNING:teuthology.provision.fog.smithi196:[Errno None] Unable to connect to port 22 on 172.21.15.196
2023-11-29T09:53:25.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-29T09:53:28.258 WARNING:teuthology.provision.fog.smithi196:[Errno None] Unable to connect to port 22 on 172.21.15.196
2023-11-29T09:53:34.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-29T09:53:37.318 WARNING:teuthology.provision.fog.smithi196:[Errno None] Unable to connect to port 22 on 172.21.15.196
2023-11-29T09:53:43.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-29T09:53:46.406 WARNING:teuthology.provision.fog.smithi196:[Errno None] Unable to connect to port 22 on 172.21.15.196
2023-11-29T09:53:48.730 WARNING:teuthology.provision.fog.smithi032:timed out
2023-11-29T09:53:52.411 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-29T09:53:54.733 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-29T09:53:54.735 WARNING:teuthology.provision.fog.smithi032:[Errno None] Unable to connect to port 22 on 172.21.15.32
2023-11-29T09:53:55.490 WARNING:teuthology.provision.fog.smithi196:[Errno None] Unable to connect to port 22 on 172.21.15.196
2023-11-29T09:54:00.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-29T09:54:00.968 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-11-29T09:54:00.969 DEBUG:teuthology.orchestra.run.smithi032:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-11-29T09:54:01.388 INFO:teuthology.provision.fog.smithi032:Node is ready
2023-11-29T09:54:01.388 DEBUG:teuthology.orchestra.run.smithi032:> hostname
2023-11-29T09:54:01.436 INFO:teuthology.orchestra.run.smithi032.stdout:smithi032.front.sepia.ceph.com
2023-11-29T09:54:01.437 DEBUG:teuthology.orchestra.run.smithi032:> grep smithi032.front.sepia.ceph.com /etc/hosts
2023-11-29T09:54:01.484 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32 smithi032.front.sepia.ceph.com smithi032
2023-11-29T09:54:01.485 DEBUG:teuthology.orchestra.run.smithi032:> sudo hostname smithi032
2023-11-29T09:54:01.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-29T09:54:01.625 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i -e 's/smithi032.front.sepia.ceph.com/smithi032/g' /etc/hosts
2023-11-29T09:54:01.640 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i -e 's/172.21.15.32/172.21.15.32/g' /etc/hosts
2023-11-29T09:54:01.695 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i -e 's/smithi032.front.sepia.ceph.com/smithi032/g' /etc/hostname
2023-11-29T09:54:01.750 DEBUG:teuthology.orchestra.run.smithi032:> sudo hostname smithi032
2023-11-29T09:54:01.804 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release
2023-11-29T09:54:01.857 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2023-11-29T09:54:01.857 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="Ubuntu"
2023-11-29T09:54:01.857 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="22.04"
2023-11-29T09:54:01.857 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2023-11-29T09:54:01.857 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_CODENAME=jammy
2023-11-29T09:54:01.857 INFO:teuthology.orchestra.run.smithi032.stdout:ID=ubuntu
2023-11-29T09:54:01.857 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE=debian
2023-11-29T09:54:01.857 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://www.ubuntu.com/"
2023-11-29T09:54:01.858 INFO:teuthology.orchestra.run.smithi032.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-11-29T09:54:01.858 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-11-29T09:54:01.858 INFO:teuthology.orchestra.run.smithi032.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-11-29T09:54:01.858 INFO:teuthology.orchestra.run.smithi032.stdout:UBUNTU_CODENAME=jammy
2023-11-29T09:54:01.858 INFO:teuthology.provision.fog.smithi032:Deploy complete!
2023-11-29T09:54:01.860 DEBUG:teuthology.parallel:result is None
2023-11-29T09:54:04.550 WARNING:teuthology.provision.fog.smithi196:[Errno None] Unable to connect to port 22 on 172.21.15.196
2023-11-29T09:54:10.554 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-29T09:55:10.617 WARNING:teuthology.provision.fog.smithi196:timed out
2023-11-29T09:55:16.623 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-29T09:55:16.763 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-11-29T09:55:16.764 DEBUG:teuthology.orchestra.run.smithi196:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-11-29T09:55:17.184 INFO:teuthology.provision.fog.smithi196:Node is ready
2023-11-29T09:55:17.185 DEBUG:teuthology.orchestra.run.smithi196:> hostname
2023-11-29T09:55:17.232 INFO:teuthology.orchestra.run.smithi196.stdout:smithi196.front.sepia.ceph.com
2023-11-29T09:55:17.232 DEBUG:teuthology.orchestra.run.smithi196:> grep smithi196.front.sepia.ceph.com /etc/hosts
2023-11-29T09:55:17.280 INFO:teuthology.orchestra.run.smithi196.stdout:172.21.15.196 smithi196.front.sepia.ceph.com smithi196
2023-11-29T09:55:17.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo hostname smithi196
2023-11-29T09:55:17.427 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i -e 's/smithi196.front.sepia.ceph.com/smithi196/g' /etc/hosts
2023-11-29T09:55:17.441 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i -e 's/172.21.15.196/172.21.15.196/g' /etc/hosts
2023-11-29T09:55:17.500 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i -e 's/smithi196.front.sepia.ceph.com/smithi196/g' /etc/hostname
2023-11-29T09:55:17.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo hostname smithi196
2023-11-29T09:55:17.618 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release
2023-11-29T09:55:17.668 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2023-11-29T09:55:17.668 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="Ubuntu"
2023-11-29T09:55:17.668 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="22.04"
2023-11-29T09:55:17.668 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2023-11-29T09:55:17.668 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_CODENAME=jammy
2023-11-29T09:55:17.668 INFO:teuthology.orchestra.run.smithi196.stdout:ID=ubuntu
2023-11-29T09:55:17.668 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE=debian
2023-11-29T09:55:17.668 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://www.ubuntu.com/"
2023-11-29T09:55:17.669 INFO:teuthology.orchestra.run.smithi196.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-11-29T09:55:17.669 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-11-29T09:55:17.669 INFO:teuthology.orchestra.run.smithi196.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-11-29T09:55:17.669 INFO:teuthology.orchestra.run.smithi196.stdout:UBUNTU_CODENAME=jammy
2023-11-29T09:55:17.669 INFO:teuthology.provision.fog.smithi196:Deploy complete!
2023-11-29T09:55:17.670 DEBUG:teuthology.parallel:result is None
2023-11-29T09:55:17.856 INFO:teuthology.lock.ops:Checking smithi032.front.sepia.ceph.com
2023-11-29T09:55:17.856 INFO:teuthology.lock.ops:New key found. Updating...
2023-11-29T09:55:17.875 INFO:teuthology.lock.ops:Checking smithi196.front.sepia.ceph.com
2023-11-29T09:55:17.875 INFO:teuthology.lock.ops:New key found. Updating...
2023-11-29T09:55:17.901 INFO:teuthology.lock.ops:Updating [smithi032.front.sepia.ceph.com]: set os type and version on server
2023-11-29T09:55:17.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-11-29T09:55:17.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-29T09:55:17.982 DEBUG:teuthology.orchestra.run.smithi032:> true
2023-11-29T09:55:18.206 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-11-29T09:55:18.206 DEBUG:teuthology.orchestra.run.smithi032:> uname -m
2023-11-29T09:55:18.252 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64
2023-11-29T09:55:18.252 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release
2023-11-29T09:55:18.300 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2023-11-29T09:55:18.300 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="Ubuntu"
2023-11-29T09:55:18.300 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="22.04"
2023-11-29T09:55:18.300 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2023-11-29T09:55:18.300 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_CODENAME=jammy
2023-11-29T09:55:18.300 INFO:teuthology.orchestra.run.smithi032.stdout:ID=ubuntu
2023-11-29T09:55:18.300 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE=debian
2023-11-29T09:55:18.300 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://www.ubuntu.com/"
2023-11-29T09:55:18.300 INFO:teuthology.orchestra.run.smithi032.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-11-29T09:55:18.301 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-11-29T09:55:18.301 INFO:teuthology.orchestra.run.smithi032.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-11-29T09:55:18.301 INFO:teuthology.orchestra.run.smithi032.stdout:UBUNTU_CODENAME=jammy
2023-11-29T09:55:18.301 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server
2023-11-29T09:55:18.333 INFO:teuthology.lock.ops:Updating [smithi196.front.sepia.ceph.com]: set os type and version on server
2023-11-29T09:55:18.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com'
2023-11-29T09:55:18.333 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-29T09:55:18.414 DEBUG:teuthology.orchestra.run.smithi196:> true
2023-11-29T09:55:18.506 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com'
2023-11-29T09:55:18.506 DEBUG:teuthology.orchestra.run.smithi196:> uname -m
2023-11-29T09:55:18.555 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64
2023-11-29T09:55:18.555 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release
2023-11-29T09:55:18.603 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2023-11-29T09:55:18.603 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="Ubuntu"
2023-11-29T09:55:18.603 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="22.04"
2023-11-29T09:55:18.603 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2023-11-29T09:55:18.603 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_CODENAME=jammy
2023-11-29T09:55:18.604 INFO:teuthology.orchestra.run.smithi196.stdout:ID=ubuntu
2023-11-29T09:55:18.604 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE=debian
2023-11-29T09:55:18.604 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://www.ubuntu.com/"
2023-11-29T09:55:18.604 INFO:teuthology.orchestra.run.smithi196.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-11-29T09:55:18.604 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-11-29T09:55:18.604 INFO:teuthology.orchestra.run.smithi196.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-11-29T09:55:18.604 INFO:teuthology.orchestra.run.smithi196.stdout:UBUNTU_CODENAME=jammy
2023-11-29T09:55:18.604 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server
2023-11-29T09:55:18.624 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T09:55:18.729 INFO:teuthology.dispatcher.supervisor:Running job 7470566
2023-11-29T09:55:18.730 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_735a43c3d8e74286f980f1d14817fb093cd8147e/virtualenv/bin/teuthology -v --owner scheduled_mkogan@teuthology --archive /home/teuthworker/archive/mkogan-2023-11-29_09:41:25-rgw-rgw-wip-t63445-valg-supp_i002-distro-default-smithi/7470566 --name mkogan-2023-11-29_09:41:25-rgw-rgw-wip-t63445-valg-supp_i002-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi} -- /home/teuthworker/archive/mkogan-2023-11-29_09:41:25-rgw-rgw-wip-t63445-valg-supp_i002-distro-default-smithi/7470566/orig.config.yaml
2023-11-29T09:55:18.740 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/mkogan-2023-11-29_09:41:25-rgw-rgw-wip-t63445-valg-supp_i002-distro-default-smithi/7470566
2023-11-29T09:55:18.740 INFO:teuthology.dispatcher.supervisor:Job PID: 2214412
2023-11-29T09:55:18.740 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-11-29T09:57:18.841 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T09:59:19.024 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:01:19.207 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:03:19.375 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:05:19.542 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:07:19.711 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:09:19.875 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:11:20.043 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:13:20.207 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:15:20.372 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:17:20.522 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:19:20.675 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:21:20.795 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:23:20.964 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:25:21.107 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:27:21.270 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:29:21.433 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:31:21.600 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:33:21.763 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:35:21.927 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:37:22.085 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:39:22.255 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:41:22.431 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:43:22.594 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:45:22.761 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:47:22.927 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:49:23.079 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:51:23.238 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:53:23.403 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:55:23.539 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:57:23.695 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T10:59:23.855 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:01:24.021 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:03:24.187 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:05:24.351 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:07:24.516 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:09:24.687 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:11:24.840 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:13:25.013 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:15:25.176 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:17:25.330 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:19:25.497 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:21:25.623 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:23:25.791 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:25:25.957 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:27:26.131 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:29:26.286 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:31:26.454 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:33:26.614 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:35:26.869 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:37:27.034 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:39:27.139 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:41:27.306 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:43:27.453 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:45:27.600 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:47:27.767 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:49:27.860 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:51:28.025 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:53:28.191 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:55:28.347 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:57:28.512 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T11:59:28.687 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:01:28.851 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:03:29.023 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:05:29.192 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:07:29.307 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:09:29.471 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:11:29.633 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:13:29.747 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:15:29.906 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:17:30.004 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:19:30.179 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:21:30.345 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:23:30.591 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:25:30.762 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:27:31.045 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:29:31.207 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:31:31.377 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:33:31.523 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:35:31.691 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:37:31.856 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:39:32.019 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:41:32.184 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:43:32.359 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:45:32.529 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:47:32.770 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:49:32.939 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:51:33.098 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:53:33.251 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:55:33.427 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:57:33.548 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T12:59:33.710 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:01:33.873 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:03:34.022 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:05:34.186 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:07:34.341 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:09:34.488 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:11:34.630 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:13:34.788 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:15:34.930 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:17:35.095 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:19:35.307 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:21:35.420 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:23:35.568 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:25:35.740 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:27:35.884 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:29:36.055 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:31:36.224 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:33:36.467 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:35:36.627 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:37:36.783 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T13:37:36.853 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1
2023-11-29T13:37:36.996 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-11-29T13:37:36.997 INFO:teuthology.nuke:Checking targets against current locks
2023-11-29T13:37:37.043 DEBUG:teuthology.nuke:shortname: smithi032
2023-11-29T13:37:37.043 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-29T13:37:37.062 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-11-29_09:41:25-rgw-rgw-wip-t63445-valg-supp_i002-distro-default-smithi/7470566', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-29 09:42:08.954575', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEncNjdCShyKHgrIQgS6EZag8tXp2yfI79j7yzQfiMGX4ulZwFvxNHdzYY6QL/wiqF8zJ7NyM2Muin80GmFokv0='}
2023-11-29T13:37:37.071 DEBUG:teuthology.nuke:shortname: smithi196
2023-11-29T13:37:37.071 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-29T13:37:37.094 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-11-29_09:41:25-rgw-rgw-wip-t63445-valg-supp_i002-distro-default-smithi/7470566', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-29 09:42:08.952978', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDECkAa8tGiA9LyQLfUlOxg2W5T7YgtPfSW10X3Y0Gnt2bk293BaVtgjMh9waNWLRRRcCDJ7ZMoJ5eP4VAni2UE='}
2023-11-29T13:37:37.103 INFO:teuthology.orchestra.console.smithi032:Power off
2023-11-29T13:37:37.103 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-29T13:37:37.124 DEBUG:teuthology.orchestra.console.smithi032:power off output: Chassis Power Control: Down/Off
2023-11-29T13:37:37.124 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-29T13:37:37.144 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off
2023-11-29T13:37:37.157 INFO:teuthology.orchestra.console.smithi196:Power off
2023-11-29T13:37:37.157 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-29T13:37:37.180 DEBUG:teuthology.orchestra.console.smithi196:power off output: Chassis Power Control: Down/Off
2023-11-29T13:37:37.180 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-29T13:37:37.200 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is off
2023-11-29T13:37:37.245 INFO:teuthology.orchestra.console.smithi032:Power off completed
2023-11-29T13:37:37.301 INFO:teuthology.orchestra.console.smithi196:Power off completed
2023-11-29T13:37:37.393 INFO:teuthology.lock.ops:unlocked: smithi032.front.sepia.ceph.com
2023-11-29T13:37:37.443 INFO:teuthology.lock.ops:unlocked: smithi196.front.sepia.ceph.com