2021-03-19T09:49:48.421 INFO:root:teuthology version: 1.1.0-b9656917 2021-03-19T09:49:48.531 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi116.front.sepia.ceph.com' reimaging 2021-03-19T09:49:48.532 INFO:teuthology.lock.ops:Updating [ubuntu@smithi116.front.sepia.ceph.com]: reset os type and version on server 2021-03-19T09:49:48.532 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2021-03-19T09:49:48.552 INFO:teuthology.lock.ops:Node 'ubuntu@smithi116.front.sepia.ceph.com' reimaging is complete 2021-03-19T09:49:48.552 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi194.front.sepia.ceph.com' reimaging 2021-03-19T09:49:48.553 INFO:teuthology.lock.ops:Updating [ubuntu@smithi194.front.sepia.ceph.com]: reset os type and version on server 2021-03-19T09:49:48.553 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2021-03-19T09:49:48.572 INFO:teuthology.lock.ops:Node 'ubuntu@smithi194.front.sepia.ceph.com' reimaging is complete 2021-03-19T09:49:48.779 INFO:teuthology.provision.fog.smithi116:Scheduling deploy of ubuntu 18.04 2021-03-19T09:49:48.782 INFO:teuthology.provision.fog.smithi194:Scheduling deploy of ubuntu 18.04 2021-03-19T09:49:49.119 INFO:teuthology.orchestra.console:Power off smithi194 2021-03-19T09:49:49.134 INFO:teuthology.orchestra.console:Power off smithi116 2021-03-19T09:49:57.519 INFO:teuthology.orchestra.console:Power off for smithi116 completed 2021-03-19T09:49:57.620 INFO:teuthology.orchestra.console:Power on smithi116 2021-03-19T09:50:01.616 INFO:teuthology.orchestra.console:Power off for smithi194 completed 2021-03-19T09:50:01.717 INFO:teuthology.orchestra.console:Power on smithi194 2021-03-19T09:50:05.977 INFO:teuthology.orchestra.console:Power on for smithi194 completed 2021-03-19T09:50:05.984 INFO:teuthology.orchestra.console:Power on for smithi116 completed 2021-03-19T09:50:06.078 INFO:teuthology.provision.fog.smithi194:Waiting for deploy to finish 2021-03-19T09:50:06.085 INFO:teuthology.provision.fog.smithi116:Waiting for deploy to finish 2021-03-19T09:55:05.039 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-19T09:55:05.399 INFO:teuthology.provision.fog.smithi116:Node is ready 2021-03-19T09:55:05.449 INFO:teuthology.orchestra.run.smithi116.stdout:smithi116.front.sepia.ceph.com 2021-03-19T09:55:05.498 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116 smithi116.front.sepia.ceph.com smithi116 2021-03-19T09:55:05.816 INFO:teuthology.provision.fog.smithi116:Deploy complete! 2021-03-19T09:55:14.047 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-19T09:55:14.334 INFO:teuthology.provision.fog.smithi194:Node is ready 2021-03-19T09:55:14.383 INFO:teuthology.orchestra.run.smithi194.stdout:smithi194.front.sepia.ceph.com 2021-03-19T09:55:14.431 INFO:teuthology.orchestra.run.smithi194.stdout:172.21.15.194 smithi194.front.sepia.ceph.com smithi194 2021-03-19T09:55:14.779 INFO:teuthology.provision.fog.smithi194:Deploy complete! 2021-03-19T09:55:14.914 INFO:teuthology.lock.ops:Checking smithi116.front.sepia.ceph.com 2021-03-19T09:55:14.915 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-19T09:55:14.930 INFO:teuthology.lock.ops:Checking smithi194.front.sepia.ceph.com 2021-03-19T09:55:14.930 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-19T09:55:14.944 INFO:teuthology.lock.ops:Updating [smithi116.front.sepia.ceph.com]: set os type and version on server 2021-03-19T09:55:14.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:55:15.313 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2021-03-19T09:55:15.361 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="Ubuntu" 2021-03-19T09:55:15.361 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-03-19T09:55:15.361 INFO:teuthology.orchestra.run.smithi116.stdout:ID=ubuntu 2021-03-19T09:55:15.361 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE=debian 2021-03-19T09:55:15.361 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-03-19T09:55:15.361 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="18.04" 2021-03-19T09:55:15.361 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-19T09:55:15.361 INFO:teuthology.orchestra.run.smithi116.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-19T09:55:15.361 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-19T09:55:15.361 INFO:teuthology.orchestra.run.smithi116.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-19T09:55:15.362 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_CODENAME=bionic 2021-03-19T09:55:15.362 INFO:teuthology.orchestra.run.smithi116.stdout:UBUNTU_CODENAME=bionic 2021-03-19T09:55:15.362 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2021-03-19T09:55:15.378 INFO:teuthology.lock.ops:Updating [smithi194.front.sepia.ceph.com]: set os type and version on server 2021-03-19T09:55:15.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T09:55:15.743 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2021-03-19T09:55:15.791 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="Ubuntu" 2021-03-19T09:55:15.791 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-03-19T09:55:15.791 INFO:teuthology.orchestra.run.smithi194.stdout:ID=ubuntu 2021-03-19T09:55:15.791 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE=debian 2021-03-19T09:55:15.791 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-03-19T09:55:15.791 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="18.04" 2021-03-19T09:55:15.792 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-19T09:55:15.792 INFO:teuthology.orchestra.run.smithi194.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-19T09:55:15.792 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-19T09:55:15.792 INFO:teuthology.orchestra.run.smithi194.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-19T09:55:15.792 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_CODENAME=bionic 2021-03-19T09:55:15.792 INFO:teuthology.orchestra.run.smithi194.stdout:UBUNTU_CODENAME=bionic 2021-03-19T09:55:15.792 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2021-03-19T09:55:15.880 INFO:teuthology.dispatcher.supervisor:Running job 5979559 2021-03-19T09:55:15.881 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/bin/teuthology -v --owner scheduled_jdillaman@dillaman-rh --archive /home/teuthworker/archive/jdillaman-2021-03-18_20:45:34-rbd-wip-jd-testing-pacific-distro-basic-smithi/5979559 --name jdillaman-2021-03-18_20:45:34-rbd-wip-jd-testing-pacific-distro-basic-smithi --description rbd/persistent-writeback-cache/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{ubuntu_latest} 4-pool/cache 5-cache-mode/rwl 6-workloads/qemu_xfstests} -- /home/teuthworker/archive/jdillaman-2021-03-18_20:45:34-rbd-wip-jd-testing-pacific-distro-basic-smithi/5979559/orig.config.yaml 2021-03-19T09:55:15.884 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/jdillaman-2021-03-18_20:45:34-rbd-wip-jd-testing-pacific-distro-basic-smithi/5979559 2021-03-19T09:55:15.884 INFO:teuthology.dispatcher.supervisor:Job PID: 20300 2021-03-19T09:55:15.884 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-03-19T16:55:44.804 ERROR:teuthology.report:Could not report results to http://paddles.front.sepia.ceph.com/ Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 498, in try_push_job_info push_job_info(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 463, in push_job_info reporter.report_job(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 320, in report_job response.raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://paddles.front.sepia.ceph.com/runs/jdillaman-2021-03-18_20:45:34-rbd-wip-jd-testing-pacific-distro-basic-smithi/jobs/5979559/ 2021-03-19T19:59:59.643 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/jdillaman-2021-03-18_20:45:34-rbd-wip-jd-testing-pacific-distro-basic-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2021-03-19T19:59:59.727 ERROR:teuthology.report:Could not report results to http://paddles.front.sepia.ceph.com/ Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 498, in try_push_job_info push_job_info(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 463, in push_job_info reporter.report_job(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/report.py", line 320, in report_job response.raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://paddles.front.sepia.ceph.com/runs/jdillaman-2021-03-18_20:45:34-rbd-wip-jd-testing-pacific-distro-basic-smithi/jobs/ 2021-03-19T21:56:07.784 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-19T21:56:08.318 INFO:teuthology.kill:Killing Pids: {20300} 2021-03-19T21:56:09.605 INFO:teuthology.kill:Nuking machines: ['smithi116', 'smithi194'] 2021-03-19T21:56:10.596 INFO:teuthology.kill:2021-03-19 21:56:10,595.595 INFO:teuthology.nuke:targets: 2021-03-19T21:56:10.597 INFO:teuthology.kill: smithi116.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOU/+B5niobvMW8DbrjOdJ80JEnuKLOQ2y1SKetNV19QqoPBEHrUOwXR0d4aW1yy72mD+emUq945t0ZRXRDLm1WipHNaPJHiNiQa75dDryctey5NiBzpaztV0snUFTZtKK5OJbWKCbDH25e+pVhI5YybDBwHtaF5RCfg1Tnoc0deRSQhIGCJ5x6ldTSbz5dmn5C8j/1MY0bvYyjxKg+2kTnseXmvGs3P4HEOk4B8VOOWCrgdLyhaq9J1IgikcbqOEVbX5UcxWQN5spBxyufIpPyCWzxoRtnwvF+dTFEH42Yc+SWKwIEzxZC2vbOs/+yEmof9rkkSiU0VzZJJOWYSTJ 2021-03-19T21:56:10.597 INFO:teuthology.kill: smithi194.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC83WWm8d2bhEWH9cCf+KLzO2KEIvAhY7XAPJVUl2TLGsiNNKeIWEGVCVdzQVAAibzr7oB34IPFV1GdKLPPpDtg/BsH7DNY+EYs0uzsiDbwso+kbfhobukWfgDv3y11vUagf6mrt4WvwmAkAla33iRG+k8mFa0k1BwXjSCCARX1POCalN9j0lZn97lLFUt6CW9yWLAUsBSVcYiL+bYxceGjtz2vTTAtOmVmuF0+/tWuSs1GyZKiRLv8Hvn/4fgWr/eWB00C+KD+pMkgQSRNG/alnX0MogFB5Gs2AEPasvxixEEHLjvgmsj1+JUkeW35N/ntW+H9qjfrIKvEJbadKMjx 2021-03-19T21:56:10.597 INFO:teuthology.kill:2021-03-19 21:56:10,596.596 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-19T21:56:10.599 INFO:teuthology.kill:2021-03-19 21:56:10,598.598 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-19T21:56:10.663 INFO:teuthology.kill:2021-03-19 21:56:10,662.662 INFO:teuthology.orchestra.console:Power off smithi116 2021-03-19T21:56:10.676 INFO:teuthology.kill:2021-03-19 21:56:10,676.676 INFO:teuthology.orchestra.console:Power off smithi194 2021-03-19T21:56:23.144 INFO:teuthology.kill:2021-03-19 21:56:23,144.144 INFO:teuthology.orchestra.console:Power off for smithi116 completed 2021-03-19T21:56:23.151 INFO:teuthology.kill:2021-03-19 21:56:23,150.150 INFO:teuthology.orchestra.console:Power off for smithi194 completed 2021-03-19T21:56:23.411 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1'] 2021-03-19T21:56:23.412 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.2', 'osd.3', 'client.0'] 2021-03-19T21:56:23.418 INFO:teuthology.misc:Compressing logs... 2021-03-19T21:56:23.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-19T21:57:23.591 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-19T21:57:23.652 INFO:teuthology.kill:No teuthology processes running 2021-03-19T21:57:24.761 INFO:teuthology.kill:Nuking machines: ['smithi116', 'smithi194'] 2021-03-19T21:57:25.789 INFO:teuthology.kill:2021-03-19 21:57:25,789.789 INFO:teuthology.nuke:targets: 2021-03-19T21:57:25.790 INFO:teuthology.kill: smithi116.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOU/+B5niobvMW8DbrjOdJ80JEnuKLOQ2y1SKetNV19QqoPBEHrUOwXR0d4aW1yy72mD+emUq945t0ZRXRDLm1WipHNaPJHiNiQa75dDryctey5NiBzpaztV0snUFTZtKK5OJbWKCbDH25e+pVhI5YybDBwHtaF5RCfg1Tnoc0deRSQhIGCJ5x6ldTSbz5dmn5C8j/1MY0bvYyjxKg+2kTnseXmvGs3P4HEOk4B8VOOWCrgdLyhaq9J1IgikcbqOEVbX5UcxWQN5spBxyufIpPyCWzxoRtnwvF+dTFEH42Yc+SWKwIEzxZC2vbOs/+yEmof9rkkSiU0VzZJJOWYSTJ 2021-03-19T21:57:25.790 INFO:teuthology.kill: smithi194.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC83WWm8d2bhEWH9cCf+KLzO2KEIvAhY7XAPJVUl2TLGsiNNKeIWEGVCVdzQVAAibzr7oB34IPFV1GdKLPPpDtg/BsH7DNY+EYs0uzsiDbwso+kbfhobukWfgDv3y11vUagf6mrt4WvwmAkAla33iRG+k8mFa0k1BwXjSCCARX1POCalN9j0lZn97lLFUt6CW9yWLAUsBSVcYiL+bYxceGjtz2vTTAtOmVmuF0+/tWuSs1GyZKiRLv8Hvn/4fgWr/eWB00C+KD+pMkgQSRNG/alnX0MogFB5Gs2AEPasvxixEEHLjvgmsj1+JUkeW35N/ntW+H9qjfrIKvEJbadKMjx 2021-03-19T21:57:25.810 INFO:teuthology.kill:2021-03-19 21:57:25,810.810 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-19T21:57:25.813 INFO:teuthology.kill:2021-03-19 21:57:25,813.813 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-19T21:57:25.874 INFO:teuthology.kill:2021-03-19 21:57:25,873.873 INFO:teuthology.orchestra.console:Power off smithi116 2021-03-19T21:57:25.887 INFO:teuthology.kill:2021-03-19 21:57:25,887.887 INFO:teuthology.orchestra.console:Power off smithi194 2021-03-19T21:57:26.018 INFO:teuthology.kill:2021-03-19 21:57:26,017.017 INFO:teuthology.orchestra.console:Power off for smithi116 completed 2021-03-19T21:57:26.029 INFO:teuthology.kill:2021-03-19 21:57:26,029.029 INFO:teuthology.orchestra.console:Power off for smithi194 completed 2021-03-19T21:57:26.161 INFO:teuthology.kill:2021-03-19 21:57:26,160.160 INFO:teuthology.lock.ops:unlocked smithi116.front.sepia.ceph.com 2021-03-19T21:57:26.169 INFO:teuthology.kill:2021-03-19 21:57:26,169.169 INFO:teuthology.lock.ops:unlocked smithi194.front.sepia.ceph.com 2021-03-19T21:59:26.476 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-03-19T21:59:26.662 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-03-19T21:59:26.666 INFO:teuthology.nuke:Checking targets against current locks 2021-03-19T21:59:26.857 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/scripts/dispatcher.py", line 33, in main teuthology.dispatcher.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 59, in main verbose File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 152, in run_job unlock_targets(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 196, in unlock_targets nuke(fake_ctx, True) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable