2021-03-22T19:27:15.823 INFO:root:teuthology version: 1.1.0-6b3150e9 2021-03-22T19:27:15.926 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi012.front.sepia.ceph.com' reimaging 2021-03-22T19:27:15.927 INFO:teuthology.lock.ops:Updating [ubuntu@smithi012.front.sepia.ceph.com]: reset os type and version on server 2021-03-22T19:27:15.927 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2021-03-22T19:27:15.944 INFO:teuthology.lock.ops:Node 'ubuntu@smithi012.front.sepia.ceph.com' reimaging is complete 2021-03-22T19:27:15.944 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi043.front.sepia.ceph.com' reimaging 2021-03-22T19:27:15.944 INFO:teuthology.lock.ops:Updating [ubuntu@smithi043.front.sepia.ceph.com]: reset os type and version on server 2021-03-22T19:27:15.945 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server 2021-03-22T19:27:15.960 INFO:teuthology.lock.ops:Node 'ubuntu@smithi043.front.sepia.ceph.com' reimaging is complete 2021-03-22T19:27:16.184 INFO:teuthology.provision.fog.smithi043:Scheduling deploy of ubuntu 20.04 2021-03-22T19:27:16.188 INFO:teuthology.provision.fog.smithi012:Scheduling deploy of ubuntu 20.04 2021-03-22T19:27:16.603 INFO:teuthology.orchestra.console:Power off smithi043 2021-03-22T19:27:16.640 INFO:teuthology.orchestra.console:Power off smithi012 2021-03-22T19:27:16.749 INFO:teuthology.orchestra.console:Power off for smithi043 completed 2021-03-22T19:27:16.784 INFO:teuthology.orchestra.console:Power off for smithi012 completed 2021-03-22T19:27:16.851 INFO:teuthology.orchestra.console:Power on smithi043 2021-03-22T19:27:16.885 INFO:teuthology.orchestra.console:Power on smithi012 2021-03-22T19:27:21.135 INFO:teuthology.orchestra.console:Power on for smithi012 completed 2021-03-22T19:27:21.236 INFO:teuthology.provision.fog.smithi012:Waiting for deploy to finish 2021-03-22T19:27:25.215 INFO:teuthology.orchestra.console:Power on for smithi043 completed 2021-03-22T19:27:25.316 INFO:teuthology.provision.fog.smithi043:Waiting for deploy to finish 2021-03-22T19:32:03.161 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-22T19:32:03.629 INFO:teuthology.provision.fog.smithi043:Node is ready 2021-03-22T19:32:03.636 INFO:teuthology.orchestra.run.smithi043.stdout:smithi043.front.sepia.ceph.com 2021-03-22T19:32:03.643 INFO:teuthology.orchestra.run.smithi043.stdout:172.21.15.43 smithi043.front.sepia.ceph.com smithi043 2021-03-22T19:32:03.829 INFO:teuthology.provision.fog.smithi043:Deploy complete! 2021-03-22T19:33:14.538 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-22T19:33:15.022 INFO:teuthology.provision.fog.smithi012:Node is ready 2021-03-22T19:33:15.029 INFO:teuthology.orchestra.run.smithi012.stdout:smithi012.front.sepia.ceph.com 2021-03-22T19:33:15.037 INFO:teuthology.orchestra.run.smithi012.stdout:172.21.15.12 smithi012.front.sepia.ceph.com smithi012 2021-03-22T19:33:15.279 INFO:teuthology.provision.fog.smithi012:Deploy complete! 2021-03-22T19:33:15.436 INFO:teuthology.lock.ops:Checking smithi012.front.sepia.ceph.com 2021-03-22T19:33:15.437 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-22T19:33:15.455 INFO:teuthology.lock.ops:Checking smithi043.front.sepia.ceph.com 2021-03-22T19:33:15.455 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-22T19:33:15.470 INFO:teuthology.lock.ops:Updating [smithi012.front.sepia.ceph.com]: set os type and version on server 2021-03-22T19:33:15.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:33:15.642 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2021-03-22T19:33:15.647 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="Ubuntu" 2021-03-22T19:33:15.647 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-03-22T19:33:15.648 INFO:teuthology.orchestra.run.smithi012.stdout:ID=ubuntu 2021-03-22T19:33:15.648 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE=debian 2021-03-22T19:33:15.648 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-03-22T19:33:15.648 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="20.04" 2021-03-22T19:33:15.649 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-22T19:33:15.649 INFO:teuthology.orchestra.run.smithi012.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-22T19:33:15.649 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-22T19:33:15.650 INFO:teuthology.orchestra.run.smithi012.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-22T19:33:15.650 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_CODENAME=focal 2021-03-22T19:33:15.650 INFO:teuthology.orchestra.run.smithi012.stdout:UBUNTU_CODENAME=focal 2021-03-22T19:33:15.652 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2021-03-22T19:33:15.668 INFO:teuthology.lock.ops:Updating [smithi043.front.sepia.ceph.com]: set os type and version on server 2021-03-22T19:33:15.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-22T19:33:15.974 INFO:teuthology.orchestra.run.smithi043.stdout:x86_64 2021-03-22T19:33:15.981 INFO:teuthology.orchestra.run.smithi043.stdout:NAME="Ubuntu" 2021-03-22T19:33:15.982 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-03-22T19:33:15.982 INFO:teuthology.orchestra.run.smithi043.stdout:ID=ubuntu 2021-03-22T19:33:15.982 INFO:teuthology.orchestra.run.smithi043.stdout:ID_LIKE=debian 2021-03-22T19:33:15.982 INFO:teuthology.orchestra.run.smithi043.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-03-22T19:33:15.983 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_ID="20.04" 2021-03-22T19:33:15.983 INFO:teuthology.orchestra.run.smithi043.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-22T19:33:15.983 INFO:teuthology.orchestra.run.smithi043.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-22T19:33:15.983 INFO:teuthology.orchestra.run.smithi043.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-22T19:33:15.983 INFO:teuthology.orchestra.run.smithi043.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-22T19:33:15.984 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_CODENAME=focal 2021-03-22T19:33:15.984 INFO:teuthology.orchestra.run.smithi043.stdout:UBUNTU_CODENAME=focal 2021-03-22T19:33:15.984 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server 2021-03-22T19:33:16.069 INFO:teuthology.dispatcher.supervisor:Running job 5988383 2021-03-22T19:33:16.070 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-03-22_16:04:17-fs-wip-yuri5-testing-2021-03-20-0828-pacific-distro-basic-smithi/5988383 --name yuriw-2021-03-22_16:04:17-fs-wip-yuri5-testing-2021-03-20-0828-pacific-distro-basic-smithi --description fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} -- /home/teuthworker/archive/yuriw-2021-03-22_16:04:17-fs-wip-yuri5-testing-2021-03-20-0828-pacific-distro-basic-smithi/5988383/orig.config.yaml 2021-03-22T19:33:16.073 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-03-22_16:04:17-fs-wip-yuri5-testing-2021-03-20-0828-pacific-distro-basic-smithi/5988383 2021-03-22T19:33:16.073 INFO:teuthology.dispatcher.supervisor:Job PID: 8704 2021-03-22T19:33:16.074 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-03-22T22:03:26.478 ERROR:teuthology.report:Could not report results to http://paddles.front.sepia.ceph.com/ Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/report.py", line 498, in try_push_job_info push_job_info(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/report.py", line 463, in push_job_info reporter.report_job(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/report.py", line 320, in report_job response.raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/virtualenv/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://paddles.front.sepia.ceph.com/runs/yuriw-2021-03-22_16:04:17-fs-wip-yuri5-testing-2021-03-20-0828-pacific-distro-basic-smithi/jobs/5988383/ 2021-03-22T22:17:27.436 ERROR:teuthology.report:Could not report results to http://paddles.front.sepia.ceph.com/ Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/report.py", line 498, in try_push_job_info push_job_info(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/report.py", line 463, in push_job_info reporter.report_job(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/report.py", line 320, in report_job response.raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/virtualenv/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://paddles.front.sepia.ceph.com/runs/yuriw-2021-03-22_16:04:17-fs-wip-yuri5-testing-2021-03-20-0828-pacific-distro-basic-smithi/jobs/5988383/ 2021-03-22T22:25:27.971 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-03-22T22:25:28.140 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-03-22T22:25:28.140 INFO:teuthology.nuke:Checking targets against current locks 2021-03-22T22:25:28.240 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-22T22:25:28.246 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-22T22:25:28.300 INFO:teuthology.orchestra.console:Power off smithi012 2021-03-22T22:25:28.314 INFO:teuthology.orchestra.console:Power off smithi043 2021-03-22T22:25:36.669 INFO:teuthology.orchestra.console:Power off for smithi012 completed 2021-03-22T22:25:36.808 ERROR:teuthology.lock.ops:failed to unlock smithi012.front.sepia.ceph.com. reason: Cannot unlock an already-unlocked node 2021-03-22T22:25:40.785 INFO:teuthology.orchestra.console:Power off for smithi043 completed 2021-03-22T22:25:40.920 ERROR:teuthology.lock.ops:failed to unlock smithi043.front.sepia.ceph.com. reason: Cannot unlock - locked_by values must match