2023-12-03T11:30:41.125 INFO:root:teuthology version: 0.0.1.dev214+gf2de108
2023-12-03T11:30:41.126 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T11:30:41.257 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fec0edbe400>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fec0e8fcd30>, signals=[15])
2023-12-03T11:30:41.257 INFO:teuthology.lock.ops:Start node 'smithi040.front.sepia.ceph.com' reimaging
2023-12-03T11:30:41.258 INFO:teuthology.lock.ops:Updating [smithi040.front.sepia.ceph.com]: reset os type and version on server
2023-12-03T11:30:41.258 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server
2023-12-03T11:30:41.283 INFO:teuthology.lock.ops:Start node 'smithi204.front.sepia.ceph.com' reimaging
2023-12-03T11:30:41.283 INFO:teuthology.lock.ops:Updating [smithi204.front.sepia.ceph.com]: reset os type and version on server
2023-12-03T11:30:41.283 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server
2023-12-03T11:30:41.398 DEBUG:teuthology.provision.fog.smithi040:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-12-03T11:30:41.419 DEBUG:teuthology.provision.fog.smithi204:Requesting image smithi_ubuntu_22.04 (ID 6)
2023-12-03T11:30:41.442 INFO:teuthology.provision.fog.smithi040:Scheduling deploy of ubuntu 22.04
2023-12-03T11:30:41.456 INFO:teuthology.provision.fog.smithi204:Scheduling deploy of ubuntu 22.04
2023-12-03T11:30:41.669 INFO:teuthology.orchestra.console.smithi204:Power off
2023-12-03T11:30:41.669 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-12-03T11:30:41.682 INFO:teuthology.orchestra.console.smithi040:Power off
2023-12-03T11:30:41.682 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-12-03T11:30:41.691 DEBUG:teuthology.orchestra.console.smithi204:power off output: Chassis Power Control: Down/Off
2023-12-03T11:30:41.692 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-03T11:30:41.703 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off
2023-12-03T11:30:41.703 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-03T11:30:41.712 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on
2023-12-03T11:30:41.724 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on
2023-12-03T11:30:45.714 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-03T11:30:45.724 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-03T11:30:45.822 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on
2023-12-03T11:30:45.832 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on
2023-12-03T11:30:49.826 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-03T11:30:49.833 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-03T11:30:49.936 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is off
2023-12-03T11:30:49.941 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on
2023-12-03T11:30:50.038 INFO:teuthology.orchestra.console.smithi204:Power off completed
2023-12-03T11:30:50.138 INFO:teuthology.orchestra.console.smithi204:Power on
2023-12-03T11:30:50.142 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-12-03T11:30:50.167 DEBUG:teuthology.orchestra.console.smithi204:power on output: Chassis Power Control: Up/On
2023-12-03T11:30:50.167 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-03T11:30:50.187 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is off
2023-12-03T11:30:53.944 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-03T11:30:54.052 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off
2023-12-03T11:30:54.153 INFO:teuthology.orchestra.console.smithi040:Power off completed
2023-12-03T11:30:54.189 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-03T11:30:54.255 INFO:teuthology.orchestra.console.smithi040:Power on
2023-12-03T11:30:54.255 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2023-12-03T11:30:54.278 DEBUG:teuthology.orchestra.console.smithi040:power on output: Chassis Power Control: Up/On
2023-12-03T11:30:54.278 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-03T11:30:54.299 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is off
2023-12-03T11:30:54.300 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off
2023-12-03T11:30:58.302 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-03T11:30:58.310 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-03T11:30:58.412 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on
2023-12-03T11:30:58.417 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on
2023-12-03T11:30:58.513 INFO:teuthology.orchestra.console.smithi204:Power on completed
2023-12-03T11:30:58.518 INFO:teuthology.orchestra.console.smithi040:Power on completed
2023-12-03T11:30:58.614 INFO:teuthology.provision.fog.smithi204:Waiting for deploy to finish
2023-12-03T11:30:58.619 INFO:teuthology.provision.fog.smithi040:Waiting for deploy to finish
2023-12-03T11:34:14.940 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-12-03T11:34:14.946 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-12-03T11:34:18.018 WARNING:teuthology.provision.fog.smithi204:[Errno None] Unable to connect to port 22 on 172.21.15.204
2023-12-03T11:34:18.022 WARNING:teuthology.provision.fog.smithi040:[Errno None] Unable to connect to port 22 on 172.21.15.40
2023-12-03T11:34:24.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-12-03T11:34:24.028 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-12-03T11:34:27.106 WARNING:teuthology.provision.fog.smithi204:[Errno None] Unable to connect to port 22 on 172.21.15.204
2023-12-03T11:34:27.107 WARNING:teuthology.provision.fog.smithi040:[Errno None] Unable to connect to port 22 on 172.21.15.40
2023-12-03T11:34:33.113 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-12-03T11:34:33.115 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-12-03T11:34:36.194 WARNING:teuthology.provision.fog.smithi204:[Errno None] Unable to connect to port 22 on 172.21.15.204
2023-12-03T11:34:36.195 WARNING:teuthology.provision.fog.smithi040:[Errno None] Unable to connect to port 22 on 172.21.15.40
2023-12-03T11:34:42.198 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-12-03T11:34:42.200 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-12-03T11:34:45.254 WARNING:teuthology.provision.fog.smithi204:[Errno None] Unable to connect to port 22 on 172.21.15.204
2023-12-03T11:34:45.255 WARNING:teuthology.provision.fog.smithi040:[Errno None] Unable to connect to port 22 on 172.21.15.40
2023-12-03T11:34:51.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-12-03T11:34:51.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-12-03T11:34:54.338 WARNING:teuthology.provision.fog.smithi040:[Errno None] Unable to connect to port 22 on 172.21.15.40
2023-12-03T11:34:54.338 WARNING:teuthology.provision.fog.smithi204:[Errno None] Unable to connect to port 22 on 172.21.15.204
2023-12-03T11:35:00.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-12-03T11:35:00.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-12-03T11:35:03.398 WARNING:teuthology.provision.fog.smithi204:[Errno None] Unable to connect to port 22 on 172.21.15.204
2023-12-03T11:35:09.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-12-03T11:36:00.395 WARNING:teuthology.provision.fog.smithi040:timed out
2023-12-03T11:36:06.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-12-03T11:36:09.406 WARNING:teuthology.provision.fog.smithi204:timed out
2023-12-03T11:36:13.933 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-12-03T11:36:13.933 DEBUG:teuthology.orchestra.run.smithi040:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-12-03T11:36:14.837 INFO:teuthology.provision.fog.smithi040:Node is ready
2023-12-03T11:36:14.837 DEBUG:teuthology.orchestra.run.smithi040:> hostname
2023-12-03T11:36:14.842 INFO:teuthology.orchestra.run.smithi040.stdout:smithi040.front.sepia.ceph.com
2023-12-03T11:36:14.842 DEBUG:teuthology.orchestra.run.smithi040:> grep smithi040.front.sepia.ceph.com /etc/hosts
2023-12-03T11:36:14.891 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40 smithi040.front.sepia.ceph.com smithi040
2023-12-03T11:36:14.891 DEBUG:teuthology.orchestra.run.smithi040:> sudo hostname smithi040
2023-12-03T11:36:15.062 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i -e 's/smithi040.front.sepia.ceph.com/smithi040/g' /etc/hosts
2023-12-03T11:36:15.077 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i -e 's/172.21.15.40/172.21.15.40/g' /etc/hosts
2023-12-03T11:36:15.132 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i -e 's/smithi040.front.sepia.ceph.com/smithi040/g' /etc/hostname
2023-12-03T11:36:15.191 DEBUG:teuthology.orchestra.run.smithi040:> sudo hostname smithi040
2023-12-03T11:36:15.246 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release
2023-12-03T11:36:15.295 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2023-12-03T11:36:15.295 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="Ubuntu"
2023-12-03T11:36:15.295 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="22.04"
2023-12-03T11:36:15.295 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2023-12-03T11:36:15.296 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_CODENAME=jammy
2023-12-03T11:36:15.296 INFO:teuthology.orchestra.run.smithi040.stdout:ID=ubuntu
2023-12-03T11:36:15.296 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE=debian
2023-12-03T11:36:15.296 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://www.ubuntu.com/"
2023-12-03T11:36:15.296 INFO:teuthology.orchestra.run.smithi040.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-12-03T11:36:15.296 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-12-03T11:36:15.297 INFO:teuthology.orchestra.run.smithi040.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-12-03T11:36:15.297 INFO:teuthology.orchestra.run.smithi040.stdout:UBUNTU_CODENAME=jammy
2023-12-03T11:36:15.297 INFO:teuthology.provision.fog.smithi040:Deploy complete!
2023-12-03T11:36:15.298 DEBUG:teuthology.parallel:result is None
2023-12-03T11:36:15.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-12-03T11:36:15.698 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-12-03T11:36:15.699 DEBUG:teuthology.orchestra.run.smithi204:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2023-12-03T11:36:16.149 INFO:teuthology.provision.fog.smithi204:Node is ready
2023-12-03T11:36:16.150 DEBUG:teuthology.orchestra.run.smithi204:> hostname
2023-12-03T11:36:16.197 INFO:teuthology.orchestra.run.smithi204.stdout:smithi204.front.sepia.ceph.com
2023-12-03T11:36:16.198 DEBUG:teuthology.orchestra.run.smithi204:> grep smithi204.front.sepia.ceph.com /etc/hosts
2023-12-03T11:36:16.246 INFO:teuthology.orchestra.run.smithi204.stdout:172.21.15.204 smithi204.front.sepia.ceph.com smithi204
2023-12-03T11:36:16.246 DEBUG:teuthology.orchestra.run.smithi204:> sudo hostname smithi204
2023-12-03T11:36:16.382 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i -e 's/smithi204.front.sepia.ceph.com/smithi204/g' /etc/hosts
2023-12-03T11:36:16.395 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i -e 's/172.21.15.204/172.21.15.204/g' /etc/hosts
2023-12-03T11:36:16.450 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i -e 's/smithi204.front.sepia.ceph.com/smithi204/g' /etc/hostname
2023-12-03T11:36:16.506 DEBUG:teuthology.orchestra.run.smithi204:> sudo hostname smithi204
2023-12-03T11:36:16.560 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release
2023-12-03T11:36:16.606 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2023-12-03T11:36:16.606 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="Ubuntu"
2023-12-03T11:36:16.606 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="22.04"
2023-12-03T11:36:16.606 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2023-12-03T11:36:16.606 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_CODENAME=jammy
2023-12-03T11:36:16.606 INFO:teuthology.orchestra.run.smithi204.stdout:ID=ubuntu
2023-12-03T11:36:16.606 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE=debian
2023-12-03T11:36:16.606 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://www.ubuntu.com/"
2023-12-03T11:36:16.606 INFO:teuthology.orchestra.run.smithi204.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-12-03T11:36:16.606 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-12-03T11:36:16.607 INFO:teuthology.orchestra.run.smithi204.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-12-03T11:36:16.607 INFO:teuthology.orchestra.run.smithi204.stdout:UBUNTU_CODENAME=jammy
2023-12-03T11:36:16.609 INFO:teuthology.provision.fog.smithi204:Deploy complete!
2023-12-03T11:36:16.609 DEBUG:teuthology.parallel:result is None
2023-12-03T11:36:16.791 INFO:teuthology.lock.ops:Checking smithi040.front.sepia.ceph.com
2023-12-03T11:36:16.792 INFO:teuthology.lock.ops:New key found. Updating...
2023-12-03T11:36:16.821 INFO:teuthology.lock.ops:Checking smithi204.front.sepia.ceph.com
2023-12-03T11:36:16.821 INFO:teuthology.lock.ops:New key found. Updating...
2023-12-03T11:36:16.838 INFO:teuthology.lock.ops:Updating [smithi040.front.sepia.ceph.com]: set os type and version on server
2023-12-03T11:36:16.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com'
2023-12-03T11:36:16.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-12-03T11:36:16.910 DEBUG:teuthology.orchestra.run.smithi040:> true
2023-12-03T11:36:17.027 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com'
2023-12-03T11:36:17.027 DEBUG:teuthology.orchestra.run.smithi040:> uname -m
2023-12-03T11:36:17.074 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64
2023-12-03T11:36:17.075 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release
2023-12-03T11:36:17.122 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2023-12-03T11:36:17.122 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="Ubuntu"
2023-12-03T11:36:17.122 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="22.04"
2023-12-03T11:36:17.122 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2023-12-03T11:36:17.122 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_CODENAME=jammy
2023-12-03T11:36:17.122 INFO:teuthology.orchestra.run.smithi040.stdout:ID=ubuntu
2023-12-03T11:36:17.123 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE=debian
2023-12-03T11:36:17.123 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://www.ubuntu.com/"
2023-12-03T11:36:17.123 INFO:teuthology.orchestra.run.smithi040.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-12-03T11:36:17.123 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-12-03T11:36:17.123 INFO:teuthology.orchestra.run.smithi040.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-12-03T11:36:17.123 INFO:teuthology.orchestra.run.smithi040.stdout:UBUNTU_CODENAME=jammy
2023-12-03T11:36:17.123 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server
2023-12-03T11:36:17.146 INFO:teuthology.lock.ops:Updating [smithi204.front.sepia.ceph.com]: set os type and version on server
2023-12-03T11:36:17.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com'
2023-12-03T11:36:17.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2023-12-03T11:36:17.228 DEBUG:teuthology.orchestra.run.smithi204:> true
2023-12-03T11:36:17.318 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com'
2023-12-03T11:36:17.318 DEBUG:teuthology.orchestra.run.smithi204:> uname -m
2023-12-03T11:36:17.369 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64
2023-12-03T11:36:17.369 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release
2023-12-03T11:36:17.413 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2023-12-03T11:36:17.413 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="Ubuntu"
2023-12-03T11:36:17.413 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="22.04"
2023-12-03T11:36:17.413 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2023-12-03T11:36:17.414 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_CODENAME=jammy
2023-12-03T11:36:17.414 INFO:teuthology.orchestra.run.smithi204.stdout:ID=ubuntu
2023-12-03T11:36:17.414 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE=debian
2023-12-03T11:36:17.414 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://www.ubuntu.com/"
2023-12-03T11:36:17.414 INFO:teuthology.orchestra.run.smithi204.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-12-03T11:36:17.414 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-12-03T11:36:17.414 INFO:teuthology.orchestra.run.smithi204.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-12-03T11:36:17.414 INFO:teuthology.orchestra.run.smithi204.stdout:UBUNTU_CODENAME=jammy
2023-12-03T11:36:17.414 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server
2023-12-03T11:36:17.438 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T11:36:17.516 INFO:teuthology.dispatcher.supervisor:Running job 7476155
2023-12-03T11:36:17.516 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/bin/teuthology -v --owner scheduled_alimasa@teuthology --archive /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476155 --name alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi} -- /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476155/orig.config.yaml
2023-12-03T11:36:17.519 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476155
2023-12-03T11:36:17.519 INFO:teuthology.dispatcher.supervisor:Job PID: 2196122
2023-12-03T11:36:17.519 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-12-03T11:38:17.618 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T11:40:17.792 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T11:42:17.947 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T11:44:18.123 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T11:46:18.343 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T11:48:18.504 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T11:50:18.669 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T11:52:18.836 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T11:54:19.010 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T11:56:19.173 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T11:58:19.331 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:00:19.497 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:02:19.648 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:04:19.813 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:06:19.938 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:08:20.099 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:10:20.266 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:12:20.426 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:14:20.588 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:16:20.752 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:18:20.903 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:20:21.144 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:22:21.306 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:24:21.472 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:26:21.647 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:28:21.808 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:30:21.971 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:32:22.139 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:34:22.302 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:36:22.456 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:38:22.614 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:40:22.763 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:42:22.927 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:44:23.094 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:46:23.268 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:48:23.437 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:50:23.606 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:52:23.755 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:54:23.939 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:56:24.104 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T12:58:24.277 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:00:24.439 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:02:24.587 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:04:24.743 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:06:24.904 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:08:24.999 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:10:25.078 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:12:25.241 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:14:25.410 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:16:25.547 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:18:25.709 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:20:25.874 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:22:26.041 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:24:26.207 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:26:26.376 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:28:26.468 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:30:26.619 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:32:26.772 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:34:26.841 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:36:27.008 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:38:27.175 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:40:27.304 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:42:27.453 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:44:27.547 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:46:27.702 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:48:27.841 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:50:27.985 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:52:28.134 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:54:28.298 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:56:28.543 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T13:58:28.703 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:00:28.824 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:02:28.942 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:04:29.084 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:06:29.143 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:08:29.271 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:10:29.435 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:12:29.585 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:14:29.748 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:16:29.916 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:18:30.094 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:20:30.255 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:22:30.421 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:24:30.583 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:26:30.749 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:28:30.923 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:30:31.103 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:32:31.271 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:34:31.443 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:36:31.607 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:38:31.755 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:40:31.920 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:42:32.083 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:44:32.253 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:46:32.420 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:48:32.600 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:50:32.756 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:52:32.923 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:54:33.084 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:56:33.251 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T14:58:33.419 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T15:00:33.611 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T15:02:33.776 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T15:04:33.935 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T15:06:34.107 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T15:08:34.287 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T15:10:34.479 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T15:12:34.641 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2023-12-03T15:12:34.858 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1
2023-12-03T15:12:35.002 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-12-03T15:12:35.003 INFO:teuthology.nuke:Checking targets against current locks
2023-12-03T15:12:35.059 DEBUG:teuthology.nuke:shortname: smithi040
2023-12-03T15:12:35.059 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-12-03T15:12:35.081 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476155', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-03 11:30:40.322273', 'locked_by': 'scheduled_alimasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMusimtOC/omwdA/dNT+iBZq73B5A1STFcKFosvCd8SzJ0PCdTaErJFWGwdSXtuSQzkjmYLSvdSYCJobWCINYBA='}
2023-12-03T15:12:35.084 DEBUG:teuthology.nuke:shortname: smithi204
2023-12-03T15:12:35.084 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-12-03T15:12:35.108 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476155', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-03 11:30:40.320481', 'locked_by': 'scheduled_alimasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFTokuna9hbob4uPpobCkw2jo0jGIjZRKuBnp4DwXn5aK3zhPPa3FWS9P101V4+UAlPX1fz9J63E3Gc/MYeSVAU='}
2023-12-03T15:12:35.129 INFO:teuthology.orchestra.console.smithi040:Power off
2023-12-03T15:12:35.130 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-12-03T15:12:35.152 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off
2023-12-03T15:12:35.153 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-03T15:12:35.168 INFO:teuthology.orchestra.console.smithi204:Power off
2023-12-03T15:12:35.168 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2023-12-03T15:12:35.177 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off
2023-12-03T15:12:35.190 DEBUG:teuthology.orchestra.console.smithi204:power off output: Chassis Power Control: Down/Off
2023-12-03T15:12:35.190 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2023-12-03T15:12:35.211 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is off
2023-12-03T15:12:35.278 INFO:teuthology.orchestra.console.smithi040:Power off completed
2023-12-03T15:12:35.313 INFO:teuthology.orchestra.console.smithi204:Power off completed
2023-12-03T15:12:35.441 INFO:teuthology.lock.ops:unlocked: smithi040.front.sepia.ceph.com
2023-12-03T15:12:35.459 INFO:teuthology.lock.ops:unlocked: smithi204.front.sepia.ceph.com