2021-03-21T16:30:48.698 INFO:root:teuthology version: 1.1.0-b6eecfe8
2021-03-21T16:30:48.745 ERROR:teuthology.report:Could not report results to http://paddles.front.sepia.ceph.com/
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/report.py", line 498, in try_push_job_info
    push_job_info(run_name, job_id, job_info)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/report.py", line 463, in push_job_info
    reporter.report_job(run_name, job_id, job_info)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/report.py", line 320, in report_job
    response.raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/virtualenv/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://paddles.front.sepia.ceph.com/runs/teuthology-2021-03-11_02:30:03-upgrade:mimic-x-nautilus-distro-basic-gibba/jobs/5955020/
2021-03-21T16:30:48.934 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba011.front.sepia.ceph.com' reimaging
2021-03-21T16:30:48.934 INFO:teuthology.lock.ops:Updating [ubuntu@gibba011.front.sepia.ceph.com]: reset os type and version on server
2021-03-21T16:30:48.935 INFO:teuthology.lock.ops:Updating gibba011.front.sepia.ceph.com on lock server
2021-03-21T16:30:48.951 INFO:teuthology.lock.ops:Node 'ubuntu@gibba011.front.sepia.ceph.com' reimaging is complete
2021-03-21T16:30:48.951 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba013.front.sepia.ceph.com' reimaging
2021-03-21T16:30:48.952 INFO:teuthology.lock.ops:Updating [ubuntu@gibba013.front.sepia.ceph.com]: reset os type and version on server
2021-03-21T16:30:48.952 INFO:teuthology.lock.ops:Updating gibba013.front.sepia.ceph.com on lock server
2021-03-21T16:30:48.969 INFO:teuthology.lock.ops:Node 'ubuntu@gibba013.front.sepia.ceph.com' reimaging is complete
2021-03-21T16:30:48.969 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba030.front.sepia.ceph.com' reimaging
2021-03-21T16:30:48.969 INFO:teuthology.lock.ops:Updating [ubuntu@gibba030.front.sepia.ceph.com]: reset os type and version on server
2021-03-21T16:30:48.969 INFO:teuthology.lock.ops:Updating gibba030.front.sepia.ceph.com on lock server
2021-03-21T16:30:48.984 INFO:teuthology.lock.ops:Node 'ubuntu@gibba030.front.sepia.ceph.com' reimaging is complete
2021-03-21T16:30:48.984 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba031.front.sepia.ceph.com' reimaging
2021-03-21T16:30:48.984 INFO:teuthology.lock.ops:Updating [ubuntu@gibba031.front.sepia.ceph.com]: reset os type and version on server
2021-03-21T16:30:48.984 INFO:teuthology.lock.ops:Updating gibba031.front.sepia.ceph.com on lock server
2021-03-21T16:30:48.998 INFO:teuthology.lock.ops:Node 'ubuntu@gibba031.front.sepia.ceph.com' reimaging is complete
2021-03-21T16:30:48.998 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba041.front.sepia.ceph.com' reimaging
2021-03-21T16:30:48.999 INFO:teuthology.lock.ops:Updating [ubuntu@gibba041.front.sepia.ceph.com]: reset os type and version on server
2021-03-21T16:30:48.999 INFO:teuthology.lock.ops:Updating gibba041.front.sepia.ceph.com on lock server
2021-03-21T16:30:49.015 INFO:teuthology.lock.ops:Node 'ubuntu@gibba041.front.sepia.ceph.com' reimaging is complete
2021-03-21T16:30:49.241 INFO:teuthology.provision.fog.gibba030:Scheduling deploy of ubuntu 16.04
2021-03-21T16:30:49.262 INFO:teuthology.provision.fog.gibba011:Scheduling deploy of ubuntu 16.04
2021-03-21T16:30:49.275 INFO:teuthology.provision.fog.gibba041:Scheduling deploy of ubuntu 16.04
2021-03-21T16:30:49.313 INFO:teuthology.provision.fog.gibba013:Scheduling deploy of ubuntu 16.04
2021-03-21T16:30:49.320 INFO:teuthology.provision.fog.gibba031:Scheduling deploy of ubuntu 16.04
2021-03-21T16:30:50.008 INFO:teuthology.orchestra.console:Power off gibba011
2021-03-21T16:30:50.044 INFO:teuthology.orchestra.console:Power off gibba030
2021-03-21T16:30:50.118 INFO:teuthology.orchestra.console:Power off gibba013
2021-03-21T16:30:50.138 INFO:teuthology.orchestra.console:Power off gibba041
2021-03-21T16:30:50.191 INFO:teuthology.orchestra.console:Power off for gibba011 completed
2021-03-21T16:30:50.197 INFO:teuthology.orchestra.console:Power off for gibba030 completed
2021-03-21T16:30:50.214 INFO:teuthology.orchestra.console:Power off gibba031
2021-03-21T16:30:50.290 INFO:teuthology.orchestra.console:Power off for gibba013 completed
2021-03-21T16:30:50.292 INFO:teuthology.orchestra.console:Power on gibba011
2021-03-21T16:30:50.303 INFO:teuthology.orchestra.console:Power on gibba030
2021-03-21T16:30:50.310 INFO:teuthology.orchestra.console:Power off for gibba041 completed
2021-03-21T16:30:50.372 INFO:teuthology.orchestra.console:Power off for gibba031 completed
2021-03-21T16:30:50.391 INFO:teuthology.orchestra.console:Power on gibba013
2021-03-21T16:30:50.412 INFO:teuthology.orchestra.console:Power on gibba041
2021-03-21T16:30:50.473 INFO:teuthology.orchestra.console:Power on gibba031
2021-03-21T16:30:58.675 INFO:teuthology.orchestra.console:Power on for gibba011 completed
2021-03-21T16:30:58.679 INFO:teuthology.orchestra.console:Power on for gibba030 completed
2021-03-21T16:30:58.772 INFO:teuthology.orchestra.console:Power on for gibba013 completed
2021-03-21T16:30:58.776 INFO:teuthology.provision.fog.gibba011:Waiting for deploy to finish
2021-03-21T16:30:58.781 INFO:teuthology.provision.fog.gibba030:Waiting for deploy to finish
2021-03-21T16:30:58.790 INFO:teuthology.orchestra.console:Power on for gibba041 completed
2021-03-21T16:30:58.846 INFO:teuthology.orchestra.console:Power on for gibba031 completed
2021-03-21T16:30:58.874 INFO:teuthology.provision.fog.gibba013:Waiting for deploy to finish
2021-03-21T16:30:58.890 INFO:teuthology.provision.fog.gibba041:Waiting for deploy to finish
2021-03-21T16:30:58.947 INFO:teuthology.provision.fog.gibba031:Waiting for deploy to finish
2021-03-21T16:33:00.879 ERROR:teuthology.orchestra.connection:Error authenticating with gibba011.front.sepia.ceph.com: Authentication failed.
2021-03-21T16:34:52.853 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104)
2021-03-21T16:34:53.769 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104)
2021-03-21T16:35:11.786 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104)
2021-03-21T16:35:13.929 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104)
2021-03-21T16:35:17.882 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-03-21T16:35:18.061 INFO:teuthology.provision.fog.gibba030:Node is ready
2021-03-21T16:35:18.103 INFO:teuthology.orchestra.run.gibba030.stdout:gibba030.front.sepia.ceph.com
2021-03-21T16:35:18.144 INFO:teuthology.orchestra.run.gibba030.stdout:172.21.2.130 gibba030.front.sepia.ceph.com gibba030
2021-03-21T16:35:18.578 INFO:teuthology.provision.fog.gibba030:Deploy complete!
2021-03-21T16:35:20.024 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-03-21T16:35:20.186 INFO:teuthology.provision.fog.gibba041:Node is ready
2021-03-21T16:35:20.226 INFO:teuthology.orchestra.run.gibba041.stdout:gibba041.front.sepia.ceph.com
2021-03-21T16:35:20.266 INFO:teuthology.orchestra.run.gibba041.stdout:172.21.2.141 gibba041.front.sepia.ceph.com gibba041
2021-03-21T16:35:20.497 INFO:teuthology.provision.fog.gibba041:Deploy complete!
2021-03-21T16:35:23.008 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-03-21T16:35:23.196 INFO:teuthology.provision.fog.gibba011:Node is ready
2021-03-21T16:35:23.239 INFO:teuthology.orchestra.run.gibba011.stdout:gibba011.front.sepia.ceph.com
2021-03-21T16:35:23.279 INFO:teuthology.orchestra.run.gibba011.stdout:172.21.2.111 gibba011.front.sepia.ceph.com gibba011
2021-03-21T16:35:23.521 INFO:teuthology.provision.fog.gibba011:Deploy complete!
2021-03-21T16:35:34.498 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-03-21T16:35:34.673 INFO:teuthology.provision.fog.gibba031:Node is ready
2021-03-21T16:35:34.714 INFO:teuthology.orchestra.run.gibba031.stdout:gibba031.front.sepia.ceph.com
2021-03-21T16:35:34.753 INFO:teuthology.orchestra.run.gibba031.stdout:172.21.2.131 gibba031.front.sepia.ceph.com gibba031
2021-03-21T16:35:34.979 INFO:teuthology.provision.fog.gibba031:Deploy complete!
2021-03-21T16:35:45.228 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104)
2021-03-21T16:36:03.341 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-03-21T16:36:03.534 INFO:teuthology.provision.fog.gibba013:Node is ready
2021-03-21T16:36:03.575 INFO:teuthology.orchestra.run.gibba013.stdout:gibba013.front.sepia.ceph.com
2021-03-21T16:36:03.615 INFO:teuthology.orchestra.run.gibba013.stdout:172.21.2.113 gibba013.front.sepia.ceph.com gibba013
2021-03-21T16:36:03.857 INFO:teuthology.provision.fog.gibba013:Deploy complete!
2021-03-21T16:36:04.040 INFO:teuthology.lock.ops:Checking gibba011.front.sepia.ceph.com
2021-03-21T16:36:04.040 INFO:teuthology.lock.ops:New key found. Updating...
2021-03-21T16:36:04.058 INFO:teuthology.lock.ops:Checking gibba013.front.sepia.ceph.com
2021-03-21T16:36:04.058 INFO:teuthology.lock.ops:New key found. Updating...
2021-03-21T16:36:04.072 INFO:teuthology.lock.ops:Checking gibba030.front.sepia.ceph.com
2021-03-21T16:36:04.072 INFO:teuthology.lock.ops:New key found. Updating...
2021-03-21T16:36:04.087 INFO:teuthology.lock.ops:Checking gibba031.front.sepia.ceph.com
2021-03-21T16:36:04.087 INFO:teuthology.lock.ops:New key found. Updating...
2021-03-21T16:36:04.102 INFO:teuthology.lock.ops:Checking gibba041.front.sepia.ceph.com
2021-03-21T16:36:04.102 INFO:teuthology.lock.ops:New key found. Updating...
2021-03-21T16:36:04.119 INFO:teuthology.lock.ops:Updating [gibba011.front.sepia.ceph.com]: set os type and version on server
2021-03-21T16:36:04.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-03-21T16:36:04.449 INFO:teuthology.orchestra.run.gibba011.stdout:x86_64
2021-03-21T16:36:04.495 INFO:teuthology.orchestra.run.gibba011.stdout:NAME="Ubuntu"
2021-03-21T16:36:04.496 INFO:teuthology.orchestra.run.gibba011.stdout:VERSION="16.04.7 LTS (Xenial Xerus)"
2021-03-21T16:36:04.496 INFO:teuthology.orchestra.run.gibba011.stdout:ID=ubuntu
2021-03-21T16:36:04.496 INFO:teuthology.orchestra.run.gibba011.stdout:ID_LIKE=debian
2021-03-21T16:36:04.496 INFO:teuthology.orchestra.run.gibba011.stdout:PRETTY_NAME="Ubuntu 16.04.7 LTS"
2021-03-21T16:36:04.496 INFO:teuthology.orchestra.run.gibba011.stdout:VERSION_ID="16.04"
2021-03-21T16:36:04.496 INFO:teuthology.orchestra.run.gibba011.stdout:HOME_URL="http://www.ubuntu.com/"
2021-03-21T16:36:04.496 INFO:teuthology.orchestra.run.gibba011.stdout:SUPPORT_URL="http://help.ubuntu.com/"
2021-03-21T16:36:04.496 INFO:teuthology.orchestra.run.gibba011.stdout:BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
2021-03-21T16:36:04.496 INFO:teuthology.orchestra.run.gibba011.stdout:VERSION_CODENAME=xenial
2021-03-21T16:36:04.496 INFO:teuthology.orchestra.run.gibba011.stdout:UBUNTU_CODENAME=xenial
2021-03-21T16:36:04.497 INFO:teuthology.lock.ops:Updating gibba011.front.sepia.ceph.com on lock server
2021-03-21T16:36:04.514 INFO:teuthology.lock.ops:Updating [gibba013.front.sepia.ceph.com]: set os type and version on server
2021-03-21T16:36:04.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-03-21T16:36:04.783 INFO:teuthology.orchestra.run.gibba013.stdout:x86_64
2021-03-21T16:36:04.823 INFO:teuthology.orchestra.run.gibba013.stdout:NAME="Ubuntu"
2021-03-21T16:36:04.823 INFO:teuthology.orchestra.run.gibba013.stdout:VERSION="16.04.7 LTS (Xenial Xerus)"
2021-03-21T16:36:04.823 INFO:teuthology.orchestra.run.gibba013.stdout:ID=ubuntu
2021-03-21T16:36:04.824 INFO:teuthology.orchestra.run.gibba013.stdout:ID_LIKE=debian
2021-03-21T16:36:04.824 INFO:teuthology.orchestra.run.gibba013.stdout:PRETTY_NAME="Ubuntu 16.04.7 LTS"
2021-03-21T16:36:04.824 INFO:teuthology.orchestra.run.gibba013.stdout:VERSION_ID="16.04"
2021-03-21T16:36:04.824 INFO:teuthology.orchestra.run.gibba013.stdout:HOME_URL="http://www.ubuntu.com/"
2021-03-21T16:36:04.824 INFO:teuthology.orchestra.run.gibba013.stdout:SUPPORT_URL="http://help.ubuntu.com/"
2021-03-21T16:36:04.824 INFO:teuthology.orchestra.run.gibba013.stdout:BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
2021-03-21T16:36:04.824 INFO:teuthology.orchestra.run.gibba013.stdout:VERSION_CODENAME=xenial
2021-03-21T16:36:04.824 INFO:teuthology.orchestra.run.gibba013.stdout:UBUNTU_CODENAME=xenial
2021-03-21T16:36:04.824 INFO:teuthology.lock.ops:Updating gibba013.front.sepia.ceph.com on lock server
2021-03-21T16:36:04.839 INFO:teuthology.lock.ops:Updating [gibba030.front.sepia.ceph.com]: set os type and version on server
2021-03-21T16:36:04.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-03-21T16:36:05.157 INFO:teuthology.orchestra.run.gibba030.stdout:x86_64
2021-03-21T16:36:05.202 INFO:teuthology.orchestra.run.gibba030.stdout:NAME="Ubuntu"
2021-03-21T16:36:05.202 INFO:teuthology.orchestra.run.gibba030.stdout:VERSION="16.04.7 LTS (Xenial Xerus)"
2021-03-21T16:36:05.202 INFO:teuthology.orchestra.run.gibba030.stdout:ID=ubuntu
2021-03-21T16:36:05.203 INFO:teuthology.orchestra.run.gibba030.stdout:ID_LIKE=debian
2021-03-21T16:36:05.203 INFO:teuthology.orchestra.run.gibba030.stdout:PRETTY_NAME="Ubuntu 16.04.7 LTS"
2021-03-21T16:36:05.203 INFO:teuthology.orchestra.run.gibba030.stdout:VERSION_ID="16.04"
2021-03-21T16:36:05.203 INFO:teuthology.orchestra.run.gibba030.stdout:HOME_URL="http://www.ubuntu.com/"
2021-03-21T16:36:05.203 INFO:teuthology.orchestra.run.gibba030.stdout:SUPPORT_URL="http://help.ubuntu.com/"
2021-03-21T16:36:05.203 INFO:teuthology.orchestra.run.gibba030.stdout:BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
2021-03-21T16:36:05.203 INFO:teuthology.orchestra.run.gibba030.stdout:VERSION_CODENAME=xenial
2021-03-21T16:36:05.203 INFO:teuthology.orchestra.run.gibba030.stdout:UBUNTU_CODENAME=xenial
2021-03-21T16:36:05.203 INFO:teuthology.lock.ops:Updating gibba030.front.sepia.ceph.com on lock server
2021-03-21T16:36:05.220 INFO:teuthology.lock.ops:Updating [gibba031.front.sepia.ceph.com]: set os type and version on server
2021-03-21T16:36:05.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-03-21T16:36:05.544 INFO:teuthology.orchestra.run.gibba031.stdout:x86_64
2021-03-21T16:36:05.588 INFO:teuthology.orchestra.run.gibba031.stdout:NAME="Ubuntu"
2021-03-21T16:36:05.589 INFO:teuthology.orchestra.run.gibba031.stdout:VERSION="16.04.7 LTS (Xenial Xerus)"
2021-03-21T16:36:05.589 INFO:teuthology.orchestra.run.gibba031.stdout:ID=ubuntu
2021-03-21T16:36:05.589 INFO:teuthology.orchestra.run.gibba031.stdout:ID_LIKE=debian
2021-03-21T16:36:05.589 INFO:teuthology.orchestra.run.gibba031.stdout:PRETTY_NAME="Ubuntu 16.04.7 LTS"
2021-03-21T16:36:05.589 INFO:teuthology.orchestra.run.gibba031.stdout:VERSION_ID="16.04"
2021-03-21T16:36:05.589 INFO:teuthology.orchestra.run.gibba031.stdout:HOME_URL="http://www.ubuntu.com/"
2021-03-21T16:36:05.589 INFO:teuthology.orchestra.run.gibba031.stdout:SUPPORT_URL="http://help.ubuntu.com/"
2021-03-21T16:36:05.589 INFO:teuthology.orchestra.run.gibba031.stdout:BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
2021-03-21T16:36:05.589 INFO:teuthology.orchestra.run.gibba031.stdout:VERSION_CODENAME=xenial
2021-03-21T16:36:05.589 INFO:teuthology.orchestra.run.gibba031.stdout:UBUNTU_CODENAME=xenial
2021-03-21T16:36:05.590 INFO:teuthology.lock.ops:Updating gibba031.front.sepia.ceph.com on lock server
2021-03-21T16:36:05.605 INFO:teuthology.lock.ops:Updating [gibba041.front.sepia.ceph.com]: set os type and version on server
2021-03-21T16:36:05.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-03-21T16:36:05.937 INFO:teuthology.orchestra.run.gibba041.stdout:x86_64
2021-03-21T16:36:05.981 INFO:teuthology.orchestra.run.gibba041.stdout:NAME="Ubuntu"
2021-03-21T16:36:05.981 INFO:teuthology.orchestra.run.gibba041.stdout:VERSION="16.04.7 LTS (Xenial Xerus)"
2021-03-21T16:36:05.981 INFO:teuthology.orchestra.run.gibba041.stdout:ID=ubuntu
2021-03-21T16:36:05.981 INFO:teuthology.orchestra.run.gibba041.stdout:ID_LIKE=debian
2021-03-21T16:36:05.981 INFO:teuthology.orchestra.run.gibba041.stdout:PRETTY_NAME="Ubuntu 16.04.7 LTS"
2021-03-21T16:36:05.982 INFO:teuthology.orchestra.run.gibba041.stdout:VERSION_ID="16.04"
2021-03-21T16:36:05.982 INFO:teuthology.orchestra.run.gibba041.stdout:HOME_URL="http://www.ubuntu.com/"
2021-03-21T16:36:05.982 INFO:teuthology.orchestra.run.gibba041.stdout:SUPPORT_URL="http://help.ubuntu.com/"
2021-03-21T16:36:05.982 INFO:teuthology.orchestra.run.gibba041.stdout:BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
2021-03-21T16:36:05.982 INFO:teuthology.orchestra.run.gibba041.stdout:VERSION_CODENAME=xenial
2021-03-21T16:36:05.982 INFO:teuthology.orchestra.run.gibba041.stdout:UBUNTU_CODENAME=xenial
2021-03-21T16:36:05.983 INFO:teuthology.lock.ops:Updating gibba041.front.sepia.ceph.com on lock server
2021-03-21T16:36:06.079 INFO:teuthology.dispatcher.supervisor:Running job 5955020
2021-03-21T16:36:06.079 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-03-11_02:30:03-upgrade:mimic-x-nautilus-distro-basic-gibba/5955020 --name teuthology-2021-03-11_02:30:03-upgrade:mimic-x-nautilus-distro-basic-gibba --description upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-nautilus 6.1-msgr2 7-final-workload/{rbd-python rgw-swift-ragweed_check snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_16.04 thrashosds-health} -- /home/teuthworker/archive/teuthology-2021-03-11_02:30:03-upgrade:mimic-x-nautilus-distro-basic-gibba/5955020/orig.config.yaml
2021-03-21T16:36:06.086 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-03-11_02:30:03-upgrade:mimic-x-nautilus-distro-basic-gibba/5955020
2021-03-21T16:36:06.086 INFO:teuthology.dispatcher.supervisor:Job PID: 32064
2021-03-21T16:36:06.086 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-03-22T03:34:20.051 ERROR:teuthology.report:Could not report results to http://paddles.front.sepia.ceph.com/
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/report.py", line 498, in try_push_job_info
    push_job_info(run_name, job_id, job_info)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/report.py", line 463, in push_job_info
    reporter.report_job(run_name, job_id, job_info)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/report.py", line 320, in report_job
    response.raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/virtualenv/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://paddles.front.sepia.ceph.com/runs/teuthology-2021-03-11_02:30:03-upgrade:mimic-x-nautilus-distro-basic-gibba/jobs/5955020/
2021-03-22T04:36:21.394 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-03-22T04:36:21.695 INFO:teuthology.kill:Killing Pids: {32064}
2021-03-22T04:36:23.465 INFO:teuthology.kill:Nuking machines: ['gibba011', 'gibba013', 'gibba030', 'gibba031', 'gibba041']
2021-03-22T04:36:24.471 INFO:teuthology.kill:2021-03-22 04:36:24,471.471 INFO:teuthology.nuke:targets:
2021-03-22T04:36:24.472 INFO:teuthology.kill:  gibba011.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDuE65mKarAhpEO3Ns/bm+0tc0+tWivynEjQ1YRku92DwgO6yR2YAKZzil2HnOEBJ9xo0vHFVYBwy4MaCTzzGsXvbovAeFSWjbwuKrw48+b3nmzvt4t3Sc8K3URw/9tGLMT4q8jijp6IzFcqASO//yDS9J4xFhEt1CUa85qtOPCgNnHglzwZI/S8WWxHcnUJnUo2zrmT+z6JaSvjqYLGKMBzVQZTprmX0RpLmwfb+bHj5i4bfyJqutkJJRKEW7iDXdBT+FwxdtZVHwrUPTY8ImaZskopGHmBPw7pbwwVLMltNXbWzv99QfdTcO1Q0F77RbjaTVe9+eZANRa4HkX6C1R
2021-03-22T04:36:24.472 INFO:teuthology.kill:  gibba013.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAWTk9DMOYJmUflvpNc6xkOL7+T+s1h5Oet5Hw0tIWvEy7lOAL7PhM2jIVAKoEN+jirff1IUd3g3ysajD0i07PT3Jd5dkgA0z4GpOtrVhcW1uycCmf1nwg36DEDhqVNgyCo8kJAqyHaRlqOZClyYNOORZ1hE3L3S8g5xXA4VpuZsDzDNtVTzSLtQsjxosLZ3KX3QJ2Ld0G6AGoI/pB2g+mjsQVJBDSH1hF7yJRAve8sWf6engWw2I4wavG2sPdRIcTOt1nPIlSn5SvmMZxJAj7KLcIA/6YuYncW6xmhpT/LHXbgJ+rDDgUJibXT6b7f+I4bP35oDrrRY/Np1hxOyKT
2021-03-22T04:36:24.472 INFO:teuthology.kill:  gibba030.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCrZC7i8FLXw04pi027t+A1uTquD/BP92eBnsS4tz3EW/EuLL2lCVtYQbsUU9Ruu16a1nzl+j6b2Oc5pgfm2Qtn9ZLfi2AS738MYSWobUb2j+eHVXUTNmjQjIBY7Y3EvA5CaTIYBZ7qXhARyEPh4cjDqnfWFIqi678Do23orMq9XGzXmt19XyAXj08hSnGoA8eagsJjW6iAflw8HnVdcEhWNabqc508/j4nkMbOy/63GCHofSABmw2ksVQMKv2s1Uo589EXuKqaxRLZoO1qb2sA+KboIc+DVLBZVqeQVU6tevetxRLJLOQzVRAW7zzWjGzK071LyVkfw+pn+1GYNpC1
2021-03-22T04:36:24.472 INFO:teuthology.kill:  gibba031.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzePoCG6VtfbqxOySzBqrbhUlYx/iLt04F2K5Uz3Vhz2ZB53l7UaWPwR/AoTCEw7G8kiiWStFb2iyjt/+W14W5YCTfwHPhHHVkHh8KmS9E10bUw8yNkWjrZSrD9Amaxse6UKvXaV2FQdAOBcQwsr29UjBbwi/s0UYQGsnOw2MgurZtEeBvgc7QhG712fSuP6ia7LFsjWu1p87iWVCG/gf9Zv4E1uIQRnh1s9F0BBYECvKeVzl01Uq0zYxwPT6nIpBqe7w+cZcEILazdyr0QEeMOHscmzhHpnSUDrVtvT0Gw9O18s1Wkmb1KLwBVJpvpQ+S/cBkjImImzw4I/+SAEVh
2021-03-22T04:36:24.472 INFO:teuthology.kill:  gibba041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgjrbkGJ2Xc3gvY0IAv6JvIp0EAdR7D6xhpV3AktrfPJ+/X9DIx+Ii1bbmsEaCFZFINCTT33BjWle4iCYVY+NKQQax4ymsvdphKTscCeyLYrEQm+doZYwgLhFGNvZtTsn2f0VwpNNZDkEIPA5phNDVLNRIFBTaDtp4RpRu+yFgNf7YY1rmpbberoabdFzs0NPhThXST4xjC8BxAc44s7QYpjp16Fu02kGx4pSRNT8F+FWQakYHmenQBL9xJQ9tBCBrpBjcUrtR0ruLzU325afrQSsYkBaMKhNWL6uuwfB+GYndwylED1OldSZWDFfQDfEr7NGDKOw/P0AieR8OuVPN
2021-03-22T04:36:24.472 INFO:teuthology.kill:2021-03-22 04:36:24,471.471 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-03-22T04:36:24.474 INFO:teuthology.kill:2021-03-22 04:36:24,474.474 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-03-22T04:36:24.475 INFO:teuthology.kill:2021-03-22 04:36:24,475.475 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-03-22T04:36:24.476 INFO:teuthology.kill:2021-03-22 04:36:24,476.476 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-03-22T04:36:24.477 INFO:teuthology.kill:2021-03-22 04:36:24,477.477 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-03-22T04:36:24.545 INFO:teuthology.kill:2021-03-22 04:36:24,545.545 INFO:teuthology.orchestra.console:Power off gibba041
2021-03-22T04:36:24.558 INFO:teuthology.kill:2021-03-22 04:36:24,558.558 INFO:teuthology.orchestra.console:Power off gibba030
2021-03-22T04:36:24.569 INFO:teuthology.kill:2021-03-22 04:36:24,569.569 INFO:teuthology.orchestra.console:Power off gibba031
2021-03-22T04:36:24.579 INFO:teuthology.kill:2021-03-22 04:36:24,579.579 INFO:teuthology.orchestra.console:Power off gibba013
2021-03-22T04:36:24.592 INFO:teuthology.kill:2021-03-22 04:36:24,591.591 INFO:teuthology.orchestra.console:Power off gibba011
2021-03-22T04:36:32.968 INFO:teuthology.kill:2021-03-22 04:36:32,968.968 INFO:teuthology.orchestra.console:Power off for gibba030 completed
2021-03-22T04:36:32.973 INFO:teuthology.kill:2021-03-22 04:36:32,972.972 INFO:teuthology.orchestra.console:Power off for gibba041 completed
2021-03-22T04:36:32.987 INFO:teuthology.kill:2021-03-22 04:36:32,987.987 INFO:teuthology.orchestra.console:Power off for gibba013 completed
2021-03-22T04:36:32.992 INFO:teuthology.kill:2021-03-22 04:36:32,992.992 INFO:teuthology.orchestra.console:Power off for gibba011 completed
2021-03-22T04:36:37.088 INFO:teuthology.kill:2021-03-22 04:36:37,087.087 INFO:teuthology.orchestra.console:Power off for gibba031 completed
2021-03-22T04:36:37.339 INFO:teuthology.task.internal:roles: ubuntu@gibba011.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
2021-03-22T04:36:37.339 INFO:teuthology.task.internal:roles: ubuntu@gibba013.front.sepia.ceph.com - ['mon.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2021-03-22T04:36:37.339 INFO:teuthology.task.internal:roles: ubuntu@gibba030.front.sepia.ceph.com - ['mon.c']
2021-03-22T04:36:37.339 INFO:teuthology.task.internal:roles: ubuntu@gibba031.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11']
2021-03-22T04:36:37.339 INFO:teuthology.task.internal:roles: ubuntu@gibba041.front.sepia.ceph.com - ['client.0']
2021-03-22T04:36:37.339 INFO:teuthology.misc:Compressing logs...
2021-03-22T04:36:37.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-03-22T04:37:37.343 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/misc.py", line 1374, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/cluster.py", line 64, in <listcomp>
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/remote.py", line 215, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/run.py", line 434, in run
    transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-03-22T04:37:37.480 INFO:teuthology.kill:No teuthology processes running
2021-03-22T04:37:39.380 INFO:teuthology.kill:Nuking machines: ['gibba011', 'gibba013', 'gibba030', 'gibba031', 'gibba041']
2021-03-22T04:37:41.138 INFO:teuthology.kill:2021-03-22 04:37:41,138.138 INFO:teuthology.nuke:targets:
2021-03-22T04:37:41.139 INFO:teuthology.kill:  gibba011.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDuE65mKarAhpEO3Ns/bm+0tc0+tWivynEjQ1YRku92DwgO6yR2YAKZzil2HnOEBJ9xo0vHFVYBwy4MaCTzzGsXvbovAeFSWjbwuKrw48+b3nmzvt4t3Sc8K3URw/9tGLMT4q8jijp6IzFcqASO//yDS9J4xFhEt1CUa85qtOPCgNnHglzwZI/S8WWxHcnUJnUo2zrmT+z6JaSvjqYLGKMBzVQZTprmX0RpLmwfb+bHj5i4bfyJqutkJJRKEW7iDXdBT+FwxdtZVHwrUPTY8ImaZskopGHmBPw7pbwwVLMltNXbWzv99QfdTcO1Q0F77RbjaTVe9+eZANRa4HkX6C1R
2021-03-22T04:37:41.139 INFO:teuthology.kill:  gibba013.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAWTk9DMOYJmUflvpNc6xkOL7+T+s1h5Oet5Hw0tIWvEy7lOAL7PhM2jIVAKoEN+jirff1IUd3g3ysajD0i07PT3Jd5dkgA0z4GpOtrVhcW1uycCmf1nwg36DEDhqVNgyCo8kJAqyHaRlqOZClyYNOORZ1hE3L3S8g5xXA4VpuZsDzDNtVTzSLtQsjxosLZ3KX3QJ2Ld0G6AGoI/pB2g+mjsQVJBDSH1hF7yJRAve8sWf6engWw2I4wavG2sPdRIcTOt1nPIlSn5SvmMZxJAj7KLcIA/6YuYncW6xmhpT/LHXbgJ+rDDgUJibXT6b7f+I4bP35oDrrRY/Np1hxOyKT
2021-03-22T04:37:41.139 INFO:teuthology.kill:  gibba030.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCrZC7i8FLXw04pi027t+A1uTquD/BP92eBnsS4tz3EW/EuLL2lCVtYQbsUU9Ruu16a1nzl+j6b2Oc5pgfm2Qtn9ZLfi2AS738MYSWobUb2j+eHVXUTNmjQjIBY7Y3EvA5CaTIYBZ7qXhARyEPh4cjDqnfWFIqi678Do23orMq9XGzXmt19XyAXj08hSnGoA8eagsJjW6iAflw8HnVdcEhWNabqc508/j4nkMbOy/63GCHofSABmw2ksVQMKv2s1Uo589EXuKqaxRLZoO1qb2sA+KboIc+DVLBZVqeQVU6tevetxRLJLOQzVRAW7zzWjGzK071LyVkfw+pn+1GYNpC1
2021-03-22T04:37:41.139 INFO:teuthology.kill:  gibba031.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzePoCG6VtfbqxOySzBqrbhUlYx/iLt04F2K5Uz3Vhz2ZB53l7UaWPwR/AoTCEw7G8kiiWStFb2iyjt/+W14W5YCTfwHPhHHVkHh8KmS9E10bUw8yNkWjrZSrD9Amaxse6UKvXaV2FQdAOBcQwsr29UjBbwi/s0UYQGsnOw2MgurZtEeBvgc7QhG712fSuP6ia7LFsjWu1p87iWVCG/gf9Zv4E1uIQRnh1s9F0BBYECvKeVzl01Uq0zYxwPT6nIpBqe7w+cZcEILazdyr0QEeMOHscmzhHpnSUDrVtvT0Gw9O18s1Wkmb1KLwBVJpvpQ+S/cBkjImImzw4I/+SAEVh
2021-03-22T04:37:41.139 INFO:teuthology.kill:  gibba041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgjrbkGJ2Xc3gvY0IAv6JvIp0EAdR7D6xhpV3AktrfPJ+/X9DIx+Ii1bbmsEaCFZFINCTT33BjWle4iCYVY+NKQQax4ymsvdphKTscCeyLYrEQm+doZYwgLhFGNvZtTsn2f0VwpNNZDkEIPA5phNDVLNRIFBTaDtp4RpRu+yFgNf7YY1rmpbberoabdFzs0NPhThXST4xjC8BxAc44s7QYpjp16Fu02kGx4pSRNT8F+FWQakYHmenQBL9xJQ9tBCBrpBjcUrtR0ruLzU325afrQSsYkBaMKhNWL6uuwfB+GYndwylED1OldSZWDFfQDfEr7NGDKOw/P0AieR8OuVPN
2021-03-22T04:37:41.169 INFO:teuthology.kill:2021-03-22 04:37:41,168.168 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-03-22T04:37:41.176 INFO:teuthology.kill:2021-03-22 04:37:41,175.175 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-03-22T04:37:41.177 INFO:teuthology.kill:2021-03-22 04:37:41,177.177 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-03-22T04:37:41.178 INFO:teuthology.kill:2021-03-22 04:37:41,178.178 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-03-22T04:37:41.179 INFO:teuthology.kill:2021-03-22 04:37:41,179.179 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-03-22T04:37:41.244 INFO:teuthology.kill:2021-03-22 04:37:41,244.244 INFO:teuthology.orchestra.console:Power off gibba031
2021-03-22T04:37:41.265 INFO:teuthology.kill:2021-03-22 04:37:41,265.265 INFO:teuthology.orchestra.console:Power off gibba013
2021-03-22T04:37:41.275 INFO:teuthology.kill:2021-03-22 04:37:41,275.275 INFO:teuthology.orchestra.console:Power off gibba011
2021-03-22T04:37:41.286 INFO:teuthology.kill:2021-03-22 04:37:41,286.286 INFO:teuthology.orchestra.console:Power off gibba041
2021-03-22T04:37:41.300 INFO:teuthology.kill:2021-03-22 04:37:41,300.300 INFO:teuthology.orchestra.console:Power off gibba030
2021-03-22T04:37:41.437 INFO:teuthology.kill:2021-03-22 04:37:41,437.437 INFO:teuthology.orchestra.console:Power off for gibba013 completed
2021-03-22T04:37:41.449 INFO:teuthology.kill:2021-03-22 04:37:41,448.448 INFO:teuthology.orchestra.console:Power off for gibba041 completed
2021-03-22T04:37:41.450 INFO:teuthology.kill:2021-03-22 04:37:41,450.450 INFO:teuthology.orchestra.console:Power off for gibba011 completed
2021-03-22T04:37:41.452 INFO:teuthology.kill:2021-03-22 04:37:41,452.452 INFO:teuthology.orchestra.console:Power off for gibba031 completed
2021-03-22T04:37:41.476 INFO:teuthology.kill:2021-03-22 04:37:41,475.475 INFO:teuthology.orchestra.console:Power off for gibba030 completed
2021-03-22T04:37:41.588 INFO:teuthology.kill:2021-03-22 04:37:41,587.587 INFO:teuthology.lock.ops:unlocked gibba013.front.sepia.ceph.com
2021-03-22T04:37:41.596 INFO:teuthology.kill:2021-03-22 04:37:41,596.596 INFO:teuthology.lock.ops:unlocked gibba041.front.sepia.ceph.com
2021-03-22T04:37:41.597 INFO:teuthology.kill:2021-03-22 04:37:41,597.597 ERROR:teuthology.lock.ops:failed to unlock gibba011.front.sepia.ceph.com. reason: 500
2021-03-22T04:37:41.603 INFO:teuthology.kill:2021-03-22 04:37:41,603.603 INFO:teuthology.lock.ops:unlocked gibba031.front.sepia.ceph.com
2021-03-22T04:37:41.608 INFO:teuthology.kill:2021-03-22 04:37:41,608.608 ERROR:teuthology.lock.ops:failed to unlock gibba030.front.sepia.ceph.com. reason: 500
2021-03-22T04:39:41.796 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-03-22T04:39:42.001 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2021-03-22T04:39:42.001 INFO:teuthology.nuke:Checking targets against current locks
2021-03-22T04:39:42.077 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/virtualenv/bin/teuthology-dispatcher", line 33, in <module>
    sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')())
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/scripts/dispatcher.py", line 33, in main
    teuthology.dispatcher.main(args)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/dispatcher/__init__.py", line 61, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/dispatcher/supervisor.py", line 59, in main
    verbose
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/dispatcher/supervisor.py", line 152, in run_job
    unlock_targets(job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/dispatcher/supervisor.py", line 196, in unlock_targets
    nuke(fake_ctx, True)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/nuke/__init__.py", line 250, in nuke
    if ctx.name not in lock['description']:
TypeError: argument of type 'NoneType' is not iterable