2024-02-07T02:15:12.149 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22
2024-02-07T02:15:12.150 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-02-07T02:15:12.232 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/yuriw-2024-02-07_00:12:44-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/jobs/ failed with status 500:
Internal Server Error
Internal Server Error
2024-02-07T02:15:13.564 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0f96cb3c10>, signals=[15])
2024-02-07T02:15:13.564 INFO:teuthology.lock.ops:Start node 'smithi032.front.sepia.ceph.com' reimaging
2024-02-07T02:15:13.564 INFO:teuthology.lock.ops:Updating [smithi032.front.sepia.ceph.com]: reset os type and version on server
2024-02-07T02:15:13.565 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server
2024-02-07T02:15:13.689 DEBUG:teuthology.provision.fog.smithi032:Requesting image smithi_ubuntu_22.04 (ID 6)
2024-02-07T02:15:13.725 INFO:teuthology.provision.fog.smithi032:Scheduling deploy of ubuntu 22.04
2024-02-07T02:15:13.993 INFO:teuthology.orchestra.console.smithi032:Power off
2024-02-07T02:15:13.993 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2024-02-07T02:15:14.015 DEBUG:teuthology.orchestra.console.smithi032:power off output: Chassis Power Control: Down/Off
2024-02-07T02:15:14.015 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-07T02:15:14.039 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off
2024-02-07T02:15:14.140 INFO:teuthology.orchestra.console.smithi032:Power off completed
2024-02-07T02:15:14.241 INFO:teuthology.orchestra.console.smithi032:Power on
2024-02-07T02:15:14.241 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2024-02-07T02:15:14.267 DEBUG:teuthology.orchestra.console.smithi032:power on output: Chassis Power Control: Up/On
2024-02-07T02:15:14.267 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-07T02:15:14.289 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off
2024-02-07T02:15:18.293 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2024-02-07T02:15:18.402 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on
2024-02-07T02:15:18.503 INFO:teuthology.orchestra.console.smithi032:Power on completed
2024-02-07T02:15:18.603 INFO:teuthology.provision.fog.smithi032:Waiting for deploy to finish
2024-02-07T02:18:19.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-07T02:18:23.068 WARNING:teuthology.provision.fog.smithi032:[Errno None] Unable to connect to port 22 on 172.21.15.32
2024-02-07T02:18:29.074 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-07T02:18:32.124 WARNING:teuthology.provision.fog.smithi032:[Errno None] Unable to connect to port 22 on 172.21.15.32
2024-02-07T02:18:38.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-07T02:18:41.180 WARNING:teuthology.provision.fog.smithi032:[Errno None] Unable to connect to port 22 on 172.21.15.32
2024-02-07T02:18:47.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-07T02:18:50.236 WARNING:teuthology.provision.fog.smithi032:[Errno None] Unable to connect to port 22 on 172.21.15.32
2024-02-07T02:18:56.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-07T02:18:59.292 WARNING:teuthology.provision.fog.smithi032:[Errno None] Unable to connect to port 22 on 172.21.15.32
2024-02-07T02:19:05.298 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-07T02:20:05.361 WARNING:teuthology.provision.fog.smithi032:timed out
2024-02-07T02:20:11.369 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-07T02:20:11.656 INFO:teuthology.orchestra.run:Running command with timeout 600
2024-02-07T02:20:11.656 DEBUG:teuthology.orchestra.run.smithi032:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done
2024-02-07T02:20:12.134 INFO:teuthology.provision.fog.smithi032:Node is ready
2024-02-07T02:20:12.134 DEBUG:teuthology.orchestra.run.smithi032:> hostname
2024-02-07T02:20:12.180 INFO:teuthology.orchestra.run.smithi032.stdout:smithi032.front.sepia.ceph.com
2024-02-07T02:20:12.180 DEBUG:teuthology.orchestra.run.smithi032:> grep smithi032.front.sepia.ceph.com /etc/hosts
2024-02-07T02:20:12.228 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32 smithi032.front.sepia.ceph.com smithi032
2024-02-07T02:20:12.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo hostname smithi032
2024-02-07T02:20:12.412 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i -e 's/smithi032.front.sepia.ceph.com/smithi032/g' /etc/hosts
2024-02-07T02:20:12.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i -e 's/172.21.15.32/172.21.15.32/g' /etc/hosts
2024-02-07T02:20:12.488 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i -e 's/smithi032.front.sepia.ceph.com/smithi032/g' /etc/hostname
2024-02-07T02:20:12.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo hostname smithi032
2024-02-07T02:20:12.606 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release
2024-02-07T02:20:12.655 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2024-02-07T02:20:12.656 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="Ubuntu"
2024-02-07T02:20:12.656 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="22.04"
2024-02-07T02:20:12.656 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2024-02-07T02:20:12.656 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_CODENAME=jammy
2024-02-07T02:20:12.656 INFO:teuthology.orchestra.run.smithi032.stdout:ID=ubuntu
2024-02-07T02:20:12.656 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE=debian
2024-02-07T02:20:12.656 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://www.ubuntu.com/"
2024-02-07T02:20:12.656 INFO:teuthology.orchestra.run.smithi032.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2024-02-07T02:20:12.656 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2024-02-07T02:20:12.656 INFO:teuthology.orchestra.run.smithi032.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2024-02-07T02:20:12.656 INFO:teuthology.orchestra.run.smithi032.stdout:UBUNTU_CODENAME=jammy
2024-02-07T02:20:12.657 INFO:teuthology.provision.fog.smithi032:Deploy complete!
2024-02-07T02:20:12.658 DEBUG:teuthology.parallel:result is None
2024-02-07T02:20:12.783 INFO:teuthology.lock.ops:Checking smithi032.front.sepia.ceph.com
2024-02-07T02:20:12.783 INFO:teuthology.lock.ops:New key found. Updating...
2024-02-07T02:20:12.808 INFO:teuthology.lock.ops:Updating [smithi032.front.sepia.ceph.com]: set os type and version on server
2024-02-07T02:20:12.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2024-02-07T02:20:12.809 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2024-02-07T02:20:12.897 DEBUG:teuthology.orchestra.run.smithi032:> true
2024-02-07T02:20:12.992 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com'
2024-02-07T02:20:12.992 DEBUG:teuthology.orchestra.run.smithi032:> uname -m
2024-02-07T02:20:13.046 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64
2024-02-07T02:20:13.158 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release
2024-02-07T02:20:13.164 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS"
2024-02-07T02:20:13.164 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="Ubuntu"
2024-02-07T02:20:13.164 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="22.04"
2024-02-07T02:20:13.164 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)"
2024-02-07T02:20:13.165 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_CODENAME=jammy
2024-02-07T02:20:13.165 INFO:teuthology.orchestra.run.smithi032.stdout:ID=ubuntu
2024-02-07T02:20:13.165 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE=debian
2024-02-07T02:20:13.165 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://www.ubuntu.com/"
2024-02-07T02:20:13.165 INFO:teuthology.orchestra.run.smithi032.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2024-02-07T02:20:13.165 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2024-02-07T02:20:13.165 INFO:teuthology.orchestra.run.smithi032.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2024-02-07T02:20:13.165 INFO:teuthology.orchestra.run.smithi032.stdout:UBUNTU_CODENAME=jammy
2024-02-07T02:20:13.165 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server
2024-02-07T02:20:13.191 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-02-07T02:20:13.303 INFO:teuthology.dispatcher.supervisor:Running job 7549634
2024-02-07T02:20:13.303 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-02-07_00:12:44-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7549634 --name yuriw-2024-02-07_00:12:44-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi --description rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} -- /home/teuthworker/archive/yuriw-2024-02-07_00:12:44-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7549634/orig.config.yaml
2024-02-07T02:20:13.307 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2024-02-07_00:12:44-rados-wip-yuri2-testing-2024-02-06-1154-distro-default-smithi/7549634
2024-02-07T02:20:13.308 INFO:teuthology.dispatcher.supervisor:Job PID: 1123367
2024-02-07T02:20:13.308 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2024-02-07T02:22:13.408 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-02-07T02:24:13.581 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-02-07T02:26:13.753 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-02-07T02:28:13.928 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-02-07T02:30:14.113 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-02-07T02:32:14.290 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-02-07T02:34:14.457 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-02-07T02:36:14.637 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-02-07T02:38:14.809 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/
2024-02-07T02:38:14.885 INFO:teuthology.dispatcher.supervisor:Success!
2024-02-07T02:38:14.966 INFO:teuthology.dispatcher.supervisor:Unlocking machines...
2024-02-07T02:38:15.014 INFO:teuthology.lock.ops:unlocked: smithi032.front.sepia.ceph.com