2023-11-28T22:55:04.654 INFO:root:teuthology version: 0.0.1.dev209+g5ae09a5
2023-11-28T22:55:04.654 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T22:55:04.784 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7f145b74cdf0>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7f145b454d30>, signals=[15])
2023-11-28T22:55:04.785 INFO:teuthology.lock.ops:Start node 'smithi077.front.sepia.ceph.com' reimaging
2023-11-28T22:55:04.785 INFO:teuthology.lock.ops:Updating [smithi077.front.sepia.ceph.com]: reset os type and version on server
2023-11-28T22:55:04.785 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server
2023-11-28T22:55:04.806 INFO:teuthology.lock.ops:Start node 'smithi104.front.sepia.ceph.com' reimaging
2023-11-28T22:55:04.806 INFO:teuthology.lock.ops:Updating [smithi104.front.sepia.ceph.com]: reset os type and version on server
2023-11-28T22:55:04.807 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server
2023-11-28T22:55:04.928 DEBUG:teuthology.provision.fog.smithi077:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-11-28T22:55:04.940 DEBUG:teuthology.provision.fog.smithi104:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-11-28T22:55:04.984 INFO:teuthology.provision.fog.smithi104:Scheduling deploy of ubuntu 22.04
2023-11-28T22:55:04.988 INFO:teuthology.provision.fog.smithi077:Scheduling deploy of ubuntu 22.04
2023-11-28T22:55:05.220 INFO:teuthology.orchestra.console.smithi077:Power off
2023-11-28T22:55:05.220 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-28T22:55:05.233 INFO:teuthology.orchestra.console.smithi104:Power off
2023-11-28T22:55:05.234 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-28T22:55:05.242 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off
2023-11-28T22:55:05.243 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-28T22:55:05.255 DEBUG:teuthology.orchestra.console.smithi104:power off output: Chassis Power Control: Down/Off
2023-11-28T22:55:05.255 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-28T22:55:05.265 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on
2023-11-28T22:55:05.275 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on
2023-11-28T22:55:09.267 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-28T22:55:09.276 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-28T22:55:09.376 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on
2023-11-28T22:55:09.384 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on
2023-11-28T22:55:13.382 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-28T22:55:13.388 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-28T22:55:13.490 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off
2023-11-28T22:55:13.496 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on
2023-11-28T22:55:13.592 INFO:teuthology.orchestra.console.smithi077:Power off completed
2023-11-28T22:55:13.692 INFO:teuthology.orchestra.console.smithi077:Power on
2023-11-28T22:55:13.693 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-28T22:55:13.713 DEBUG:teuthology.orchestra.console.smithi077:power on output: Chassis Power Control: Up/On
2023-11-28T22:55:13.713 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-28T22:55:13.735 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off
2023-11-28T22:55:17.501 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-28T22:55:17.609 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is off
2023-11-28T22:55:17.710 INFO:teuthology.orchestra.console.smithi104:Power off completed
2023-11-28T22:55:17.735 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-28T22:55:17.811 INFO:teuthology.orchestra.console.smithi104:Power on
2023-11-28T22:55:17.811 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-11-28T22:55:17.832 DEBUG:teuthology.orchestra.console.smithi104:power on output: Chassis Power Control: Up/On
2023-11-28T22:55:17.833 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-28T22:55:17.844 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off
2023-11-28T22:55:17.856 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is off
2023-11-28T22:55:21.848 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-28T22:55:21.857 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-28T22:55:21.966 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is off
2023-11-28T22:55:22.074 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on
2023-11-28T22:55:22.174 INFO:teuthology.orchestra.console.smithi077:Power on completed
2023-11-28T22:55:22.275 INFO:teuthology.provision.fog.smithi077:Waiting for deploy to finish
2023-11-28T22:55:25.970 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-28T22:55:26.079 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on
2023-11-28T22:55:26.179 INFO:teuthology.orchestra.console.smithi104:Power on completed
2023-11-28T22:55:26.280 INFO:teuthology.provision.fog.smithi104:Waiting for deploy to finish
2023-11-28T22:58:38.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-28T22:58:42.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-28T22:58:56.742 WARNING:teuthology.provision.fog.smithi077:[Errno None] Unable to connect to port 22 on 172.21.15.77
2023-11-28T22:59:00.578 WARNING:teuthology.provision.fog.smithi104:[Errno None] Unable to connect to port 22 on 172.21.15.104
2023-11-28T22:59:02.746 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-28T22:59:05.826 WARNING:teuthology.provision.fog.smithi077:[Errno None] Unable to connect to port 22 on 172.21.15.77
2023-11-28T22:59:06.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-28T22:59:09.634 WARNING:teuthology.provision.fog.smithi104:[Errno None] Unable to connect to port 22 on 172.21.15.104
2023-11-28T22:59:11.829 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-28T22:59:14.882 WARNING:teuthology.provision.fog.smithi077:[Errno None] Unable to connect to port 22 on 172.21.15.77
2023-11-28T22:59:15.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-28T22:59:18.690 WARNING:teuthology.provision.fog.smithi104:[Errno None] Unable to connect to port 22 on 172.21.15.104
2023-11-28T22:59:20.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-28T22:59:23.938 WARNING:teuthology.provision.fog.smithi077:[Errno None] Unable to connect to port 22 on 172.21.15.77
2023-11-28T22:59:24.692 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-28T22:59:27.746 WARNING:teuthology.provision.fog.smithi104:[Errno None] Unable to connect to port 22 on 172.21.15.104
2023-11-28T22:59:29.942 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-28T22:59:33.750 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-28T23:00:29.999 WARNING:teuthology.provision.fog.smithi077:timed out
2023-11-28T23:00:33.754 WARNING:teuthology.provision.fog.smithi104:timed out
2023-11-28T23:00:36.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-28T23:00:36.218 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-11-28T23:00:36.218 DEBUG:teuthology.orchestra.run.smithi077:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-11-28T23:00:36.662 INFO:teuthology.provision.fog.smithi077:Node is ready
2023-11-28T23:00:36.662 DEBUG:teuthology.orchestra.run.smithi077:> hostname
2023-11-28T23:00:36.711 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077.front.sepia.ceph.com
2023-11-28T23:00:36.711 DEBUG:teuthology.orchestra.run.smithi077:> grep smithi077.front.sepia.ceph.com /etc/hosts
2023-11-28T23:00:36.759 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77 smithi077.front.sepia.ceph.com smithi077
2023-11-28T23:00:36.759 DEBUG:teuthology.orchestra.run.smithi077:> sudo hostname smithi077
2023-11-28T23:00:36.898 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i -e 's/smithi077.front.sepia.ceph.com/smithi077/g' /etc/hosts
2023-11-28T23:00:36.913 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i -e 's/172.21.15.77/172.21.15.77/g' /etc/hosts
2023-11-28T23:00:36.966 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i -e 's/smithi077.front.sepia.ceph.com/smithi077/g' /etc/hostname
2023-11-28T23:00:37.023 DEBUG:teuthology.orchestra.run.smithi077:> sudo hostname smithi077
2023-11-28T23:00:37.082 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release
2023-11-28T23:00:37.131 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2023-11-28T23:00:37.131 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="Ubuntu"
2023-11-28T23:00:37.132 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="22.04"
2023-11-28T23:00:37.132 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2023-11-28T23:00:37.132 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_CODENAME=jammy
2023-11-28T23:00:37.132 INFO:teuthology.orchestra.run.smithi077.stdout:ID=ubuntu
2023-11-28T23:00:37.132 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE=debian
2023-11-28T23:00:37.132 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://www.ubuntu.com/"
2023-11-28T23:00:37.132 INFO:teuthology.orchestra.run.smithi077.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-11-28T23:00:37.132 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-11-28T23:00:37.132 INFO:teuthology.orchestra.run.smithi077.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-11-28T23:00:37.132 INFO:teuthology.orchestra.run.smithi077.stdout:UBUNTU_CODENAME=jammy
2023-11-28T23:00:37.133 INFO:teuthology.provision.fog.smithi077:Deploy complete!
2023-11-28T23:00:37.134 DEBUG:teuthology.parallel:result is None
2023-11-28T23:00:39.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-28T23:00:40.030 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-11-28T23:00:40.030 DEBUG:teuthology.orchestra.run.smithi104:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-11-28T23:00:40.463 INFO:teuthology.provision.fog.smithi104:Node is ready
2023-11-28T23:00:40.463 DEBUG:teuthology.orchestra.run.smithi104:> hostname
2023-11-28T23:00:40.509 INFO:teuthology.orchestra.run.smithi104.stdout:smithi104.front.sepia.ceph.com
2023-11-28T23:00:40.509 DEBUG:teuthology.orchestra.run.smithi104:> grep smithi104.front.sepia.ceph.com /etc/hosts
2023-11-28T23:00:40.557 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104 smithi104.front.sepia.ceph.com smithi104
2023-11-28T23:00:40.557 DEBUG:teuthology.orchestra.run.smithi104:> sudo hostname smithi104
2023-11-28T23:00:40.729 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i -e 's/smithi104.front.sepia.ceph.com/smithi104/g' /etc/hosts
2023-11-28T23:00:40.743 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i -e 's/172.21.15.104/172.21.15.104/g' /etc/hosts
2023-11-28T23:00:40.798 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i -e 's/smithi104.front.sepia.ceph.com/smithi104/g' /etc/hostname
2023-11-28T23:00:40.857 DEBUG:teuthology.orchestra.run.smithi104:> sudo hostname smithi104
2023-11-28T23:00:40.916 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release
2023-11-28T23:00:40.961 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2023-11-28T23:00:40.961 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="Ubuntu"
2023-11-28T23:00:40.962 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="22.04"
2023-11-28T23:00:40.962 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2023-11-28T23:00:40.962 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_CODENAME=jammy
2023-11-28T23:00:40.962 INFO:teuthology.orchestra.run.smithi104.stdout:ID=ubuntu
2023-11-28T23:00:40.963 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE=debian
2023-11-28T23:00:40.963 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://www.ubuntu.com/"
2023-11-28T23:00:40.963 INFO:teuthology.orchestra.run.smithi104.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-11-28T23:00:40.963 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-11-28T23:00:40.963 INFO:teuthology.orchestra.run.smithi104.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-11-28T23:00:40.963 INFO:teuthology.orchestra.run.smithi104.stdout:UBUNTU_CODENAME=jammy
2023-11-28T23:00:40.964 INFO:teuthology.provision.fog.smithi104:Deploy complete!
2023-11-28T23:00:40.964 DEBUG:teuthology.parallel:result is None
2023-11-28T23:00:41.141 INFO:teuthology.lock.ops:Checking smithi077.front.sepia.ceph.com
2023-11-28T23:00:41.142 INFO:teuthology.lock.ops:New key found. Updating...
2023-11-28T23:00:41.166 INFO:teuthology.lock.ops:Checking smithi104.front.sepia.ceph.com
2023-11-28T23:00:41.166 INFO:teuthology.lock.ops:New key found. Updating...
2023-11-28T23:00:41.183 INFO:teuthology.lock.ops:Updating [smithi077.front.sepia.ceph.com]: set os type and version on server
2023-11-28T23:00:41.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com'
2023-11-28T23:00:41.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-28T23:00:41.262 DEBUG:teuthology.orchestra.run.smithi077:> true
2023-11-28T23:00:41.382 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com'
2023-11-28T23:00:41.382 DEBUG:teuthology.orchestra.run.smithi077:> uname -m
2023-11-28T23:00:41.431 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64
2023-11-28T23:00:41.431 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release
2023-11-28T23:00:41.481 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2023-11-28T23:00:41.481 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="Ubuntu"
2023-11-28T23:00:41.482 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="22.04"
2023-11-28T23:00:41.482 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2023-11-28T23:00:41.482 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_CODENAME=jammy
2023-11-28T23:00:41.482 INFO:teuthology.orchestra.run.smithi077.stdout:ID=ubuntu
2023-11-28T23:00:41.482 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE=debian
2023-11-28T23:00:41.482 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://www.ubuntu.com/"
2023-11-28T23:00:41.482 INFO:teuthology.orchestra.run.smithi077.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-11-28T23:00:41.482 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-11-28T23:00:41.482 INFO:teuthology.orchestra.run.smithi077.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-11-28T23:00:41.482 INFO:teuthology.orchestra.run.smithi077.stdout:UBUNTU_CODENAME=jammy
2023-11-28T23:00:41.483 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server
2023-11-28T23:00:41.508 INFO:teuthology.lock.ops:Updating [smithi104.front.sepia.ceph.com]: set os type and version on server
2023-11-28T23:00:41.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com'
2023-11-28T23:00:41.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-11-28T23:00:41.584 DEBUG:teuthology.orchestra.run.smithi104:> true
2023-11-28T23:00:41.682 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com'
2023-11-28T23:00:41.682 DEBUG:teuthology.orchestra.run.smithi104:> uname -m
2023-11-28T23:00:41.733 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64
2023-11-28T23:00:41.733 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release
2023-11-28T23:00:41.781 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2023-11-28T23:00:41.781 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="Ubuntu"
2023-11-28T23:00:41.781 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="22.04"
2023-11-28T23:00:41.781 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2023-11-28T23:00:41.781 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_CODENAME=jammy
2023-11-28T23:00:41.781 INFO:teuthology.orchestra.run.smithi104.stdout:ID=ubuntu
2023-11-28T23:00:41.781 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE=debian
2023-11-28T23:00:41.781 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://www.ubuntu.com/"
2023-11-28T23:00:41.781 INFO:teuthology.orchestra.run.smithi104.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-11-28T23:00:41.782 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-11-28T23:00:41.782 INFO:teuthology.orchestra.run.smithi104.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-11-28T23:00:41.782 INFO:teuthology.orchestra.run.smithi104.stdout:UBUNTU_CODENAME=jammy
2023-11-28T23:00:41.782 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server
2023-11-28T23:00:41.799 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:00:41.890 INFO:teuthology.dispatcher.supervisor:Running job 7470234
2023-11-28T23:00:41.890 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_5ae09a5cc7064ca2bf994df19f3006b79794789a/virtualenv/bin/teuthology -v --owner scheduled_mkogan@teuthology --archive /home/teuthworker/archive/mkogan-2023-11-28_16:18:11-rgw-rgw-wip-t63445-valg-supp_i002-distro-default-smithi/7470234 --name mkogan-2023-11-28_16:18:11-rgw-rgw-wip-t63445-valg-supp_i002-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi} -- /home/teuthworker/archive/mkogan-2023-11-28_16:18:11-rgw-rgw-wip-t63445-valg-supp_i002-distro-default-smithi/7470234/orig.config.yaml
2023-11-28T23:00:41.893 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/mkogan-2023-11-28_16:18:11-rgw-rgw-wip-t63445-valg-supp_i002-distro-default-smithi/7470234
2023-11-28T23:00:41.894 INFO:teuthology.dispatcher.supervisor:Job PID: 3849540
2023-11-28T23:00:41.894 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-11-28T23:02:41.995 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:04:42.177 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:06:42.351 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:08:42.507 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:10:42.675 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:12:42.842 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:14:43.015 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:16:43.151 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:18:43.315 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:20:43.479 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:22:43.655 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:24:43.815 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:26:43.994 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:28:44.164 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:30:44.346 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:32:44.503 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:34:44.607 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:36:44.779 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:38:44.947 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:40:45.123 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:42:45.274 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:44:45.426 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:46:45.591 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:48:45.758 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:50:45.902 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:52:46.050 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:54:46.213 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:56:46.379 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-28T23:58:46.551 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:00:46.715 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:02:46.880 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:04:47.043 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:06:47.202 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:08:47.364 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:10:47.524 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:12:47.699 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:14:47.821 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:16:47.974 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:18:48.147 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:20:48.275 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:22:54.236 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:24:54.403 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:26:54.579 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:28:54.714 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:30:54.888 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:32:55.135 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:34:55.298 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:36:55.445 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:38:55.915 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:40:56.047 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:42:56.221 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:44:56.384 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:46:56.549 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:48:56.666 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:50:59.548 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:52:59.708 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:54:59.790 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:56:59.955 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T00:59:00.067 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:01:00.205 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:03:00.378 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:05:00.506 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:07:00.661 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:09:00.750 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:11:00.833 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:13:01.058 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:15:01.319 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:17:01.458 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:19:01.605 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:21:01.777 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:23:01.921 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:25:02.098 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:27:02.237 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:29:02.410 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:31:02.580 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:33:02.732 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:35:02.898 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:37:03.062 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:39:03.226 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:41:03.440 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:43:03.688 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:45:03.853 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:47:04.026 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:49:04.841 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:51:05.056 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:53:05.219 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:55:05.381 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:57:05.541 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T01:59:05.697 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T02:01:05.841 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T02:03:06.061 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T02:05:06.129 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T02:07:06.274 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T02:09:06.380 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T02:11:06.550 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T02:13:06.669 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T02:15:06.832 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T02:17:06.975 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T02:19:07.120 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T02:21:07.291 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T02:23:07.396 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T02:25:07.554 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T02:27:07.651 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-11-29T02:27:07.722 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1
2023-11-29T02:27:07.863 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-11-29T02:27:07.864 INFO:teuthology.nuke:Checking targets against current locks
2023-11-29T02:27:07.915 DEBUG:teuthology.nuke:shortname: smithi077
2023-11-29T02:27:07.915 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-29T02:27:07.938 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-11-28_16:18:11-rgw-rgw-wip-t63445-valg-supp_i002-distro-default-smithi/7470234', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 22:55:03.867555', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNw2NOEUxGBEAoQJ++ea0QY8QT7QAyNOVmeiNSKxu0aFM0Cc6+2pW7Qb4CgExBiJqodP/+d6GkCTTMlbqyNVHS8='}
2023-11-29T02:27:07.940 DEBUG:teuthology.nuke:shortname: smithi104
2023-11-29T02:27:07.941 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-11-29T02:27:07.961 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mkogan-2023-11-28_16:18:11-rgw-rgw-wip-t63445-valg-supp_i002-distro-default-smithi/7470234', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 22:55:03.869543', 'locked_by': 'scheduled_mkogan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOChyfr0KsAvi4zm04bQRjq0jDuqbQN2SdeoKqHWrn7dF5OuNIeV0fj9ZSWQ3NMPAPFZJtTs6zni6eSFKrZxmwM='}
2023-11-29T02:27:07.989 INFO:teuthology.orchestra.console.smithi077:Power off
2023-11-29T02:27:07.990 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-29T02:27:08.009 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off
2023-11-29T02:27:08.009 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-29T02:27:08.021 INFO:teuthology.orchestra.console.smithi104:Power off
2023-11-29T02:27:08.021 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-11-29T02:27:08.029 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off
2023-11-29T02:27:08.040 DEBUG:teuthology.orchestra.console.smithi104:power off output: Chassis Power Control: Down/Off
2023-11-29T02:27:08.040 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-11-29T02:27:08.059 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is off
2023-11-29T02:27:08.131 INFO:teuthology.orchestra.console.smithi077:Power off completed
2023-11-29T02:27:08.160 INFO:teuthology.orchestra.console.smithi104:Power off completed
2023-11-29T02:27:08.288 INFO:teuthology.lock.ops:unlocked: smithi077.front.sepia.ceph.com
2023-11-29T02:27:08.311 INFO:teuthology.lock.ops:unlocked: smithi104.front.sepia.ceph.com