2021-08-11T14:49:08.863 INFO:root:teuthology version: 1.1.0-321319b1 2021-08-11T14:49:08.962 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi061.front.sepia.ceph.com' reimaging 2021-08-11T14:49:08.963 INFO:teuthology.lock.ops:Updating [ubuntu@smithi061.front.sepia.ceph.com]: reset os type and version on server 2021-08-11T14:49:08.963 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server 2021-08-11T14:49:08.981 INFO:teuthology.lock.ops:Node 'ubuntu@smithi061.front.sepia.ceph.com' reimaging is complete 2021-08-11T14:49:08.981 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi099.front.sepia.ceph.com' reimaging 2021-08-11T14:49:08.982 INFO:teuthology.lock.ops:Updating [ubuntu@smithi099.front.sepia.ceph.com]: reset os type and version on server 2021-08-11T14:49:08.983 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2021-08-11T14:49:09.001 INFO:teuthology.lock.ops:Node 'ubuntu@smithi099.front.sepia.ceph.com' reimaging is complete 2021-08-11T14:49:09.193 INFO:teuthology.provision.fog.smithi061:Scheduling deploy of centos 8.2 2021-08-11T14:49:09.200 INFO:teuthology.provision.fog.smithi099:Scheduling deploy of centos 8.2 2021-08-11T14:49:09.510 INFO:teuthology.orchestra.console:Power off smithi099 2021-08-11T14:49:09.525 INFO:teuthology.orchestra.console:Power off smithi061 2021-08-11T14:49:17.881 INFO:teuthology.orchestra.console:Power off for smithi099 completed 2021-08-11T14:49:17.889 INFO:teuthology.orchestra.console:Power off for smithi061 completed 2021-08-11T14:49:17.984 INFO:teuthology.orchestra.console:Power on smithi099 2021-08-11T14:49:17.992 INFO:teuthology.orchestra.console:Power on smithi061 2021-08-11T14:49:22.243 INFO:teuthology.orchestra.console:Power on for smithi099 completed 2021-08-11T14:49:22.345 INFO:teuthology.provision.fog.smithi099:Waiting for deploy to finish 2021-08-11T14:49:26.355 INFO:teuthology.orchestra.console:Power on for smithi061 completed 2021-08-11T14:49:26.456 INFO:teuthology.provision.fog.smithi061:Waiting for deploy to finish 2021-08-11T14:53:34.867 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-11T14:53:55.237 INFO:teuthology.provision.fog.smithi099:Node is ready 2021-08-11T14:53:55.257 INFO:teuthology.orchestra.run.smithi099.stdout:smithi099.front.sepia.ceph.com 2021-08-11T14:53:55.313 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.15.99 smithi099.front.sepia.ceph.com smithi099 2021-08-11T14:53:55.651 INFO:teuthology.provision.fog.smithi099:Deploy complete! 2021-08-11T14:54:13.223 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-11T14:54:13.553 INFO:teuthology.provision.fog.smithi061:Node is ready 2021-08-11T14:54:13.569 INFO:teuthology.orchestra.run.smithi061.stdout:smithi061.front.sepia.ceph.com 2021-08-11T14:54:13.627 INFO:teuthology.orchestra.run.smithi061.stdout:172.21.15.61 smithi061.front.sepia.ceph.com smithi061 2021-08-11T14:54:13.961 INFO:teuthology.provision.fog.smithi061:Deploy complete! 2021-08-11T14:54:14.110 INFO:teuthology.lock.ops:Checking smithi061.front.sepia.ceph.com 2021-08-11T14:54:14.111 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-11T14:54:14.127 INFO:teuthology.lock.ops:Checking smithi099.front.sepia.ceph.com 2021-08-11T14:54:14.128 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-11T14:54:14.147 INFO:teuthology.lock.ops:Updating [smithi061.front.sepia.ceph.com]: set os type and version on server 2021-08-11T14:54:14.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-11T14:54:14.412 INFO:teuthology.orchestra.run.smithi061.stdout:x86_64 2021-08-11T14:54:14.466 INFO:teuthology.orchestra.run.smithi061.stdout:NAME="CentOS Linux" 2021-08-11T14:54:14.466 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION="8 (Core)" 2021-08-11T14:54:14.467 INFO:teuthology.orchestra.run.smithi061.stdout:ID="centos" 2021-08-11T14:54:14.467 INFO:teuthology.orchestra.run.smithi061.stdout:ID_LIKE="rhel fedora" 2021-08-11T14:54:14.467 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_ID="8" 2021-08-11T14:54:14.467 INFO:teuthology.orchestra.run.smithi061.stdout:PLATFORM_ID="platform:el8" 2021-08-11T14:54:14.467 INFO:teuthology.orchestra.run.smithi061.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-08-11T14:54:14.468 INFO:teuthology.orchestra.run.smithi061.stdout:ANSI_COLOR="0;31" 2021-08-11T14:54:14.468 INFO:teuthology.orchestra.run.smithi061.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-11T14:54:14.468 INFO:teuthology.orchestra.run.smithi061.stdout:HOME_URL="https://www.centos.org/" 2021-08-11T14:54:14.468 INFO:teuthology.orchestra.run.smithi061.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-08-11T14:54:14.469 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-08-11T14:54:14.469 INFO:teuthology.orchestra.run.smithi061.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-08-11T14:54:14.469 INFO:teuthology.orchestra.run.smithi061.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-08-11T14:54:14.469 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-08-11T14:54:14.469 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-08-11T14:54:14.470 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-08-11T14:54:14.471 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server 2021-08-11T14:54:14.486 INFO:teuthology.lock.ops:Updating [smithi099.front.sepia.ceph.com]: set os type and version on server 2021-08-11T14:54:14.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-11T14:54:14.762 INFO:teuthology.orchestra.run.smithi099.stdout:x86_64 2021-08-11T14:54:14.816 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="CentOS Linux" 2021-08-11T14:54:14.816 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="8 (Core)" 2021-08-11T14:54:14.816 INFO:teuthology.orchestra.run.smithi099.stdout:ID="centos" 2021-08-11T14:54:14.817 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE="rhel fedora" 2021-08-11T14:54:14.817 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="8" 2021-08-11T14:54:14.817 INFO:teuthology.orchestra.run.smithi099.stdout:PLATFORM_ID="platform:el8" 2021-08-11T14:54:14.818 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-08-11T14:54:14.818 INFO:teuthology.orchestra.run.smithi099.stdout:ANSI_COLOR="0;31" 2021-08-11T14:54:14.818 INFO:teuthology.orchestra.run.smithi099.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-11T14:54:14.818 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://www.centos.org/" 2021-08-11T14:54:14.819 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-08-11T14:54:14.819 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-08-11T14:54:14.819 INFO:teuthology.orchestra.run.smithi099.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-08-11T14:54:14.819 INFO:teuthology.orchestra.run.smithi099.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-08-11T14:54:14.819 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-08-11T14:54:14.820 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-08-11T14:54:14.820 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-08-11T14:54:14.821 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2021-08-11T14:54:14.897 INFO:teuthology.dispatcher.supervisor:Running job 6334660 2021-08-11T14:54:14.897 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-08-11_14:15:52-rados-wip-yuri3-testing-2021-08-09-1006-distro-basic-smithi/6334660 --name yuriw-2021-08-11_14:15:52-rados-wip-yuri3-testing-2021-08-09-1006-distro-basic-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-centos_8.2_kubic_stable 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/yuriw-2021-08-11_14:15:52-rados-wip-yuri3-testing-2021-08-09-1006-distro-basic-smithi/6334660/orig.config.yaml 2021-08-11T14:54:14.901 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-08-11_14:15:52-rados-wip-yuri3-testing-2021-08-09-1006-distro-basic-smithi/6334660 2021-08-11T14:54:14.902 INFO:teuthology.dispatcher.supervisor:Job PID: 18499 2021-08-11T14:54:14.902 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-08-11T22:20:46.686 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2021-08-11_14:15:52-rados-wip-yuri3-testing-2021-08-09-1006-distro-basic-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2021-08-12T02:55:06.667 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-08-12T02:55:06.918 INFO:teuthology.kill:Killing Pids: {18499} 2021-08-12T02:55:08.140 INFO:teuthology.kill:Nuking machines: ['smithi061', 'smithi099'] 2021-08-12T02:55:09.164 INFO:teuthology.kill:2021-08-12 02:55:09,164.164 INFO:teuthology.nuke:targets: 2021-08-12T02:55:09.166 INFO:teuthology.kill: smithi061.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq+vxlSRFBYMpDSSSrn1Asq2MQyLtJF4TPx8YTSsK9A4Nw+HeFd/V65RD3iyxnYdC2fm/lnnxDvf3EGh1P2Bp2kuqdo8nzpor9i0yuTfzy5U6aFqkie+tDzUL8e9y+nTbqJEiWGtOqvjwWBQcu8Hf6vOrnqInHDmzbGHDzguRv1aihH3rz1JMVOMZzC7+l34mva76Ow2Lj+/Vn4Unsk/LySUly6kO9Ois9+WUcf8oAMOmjL5jNg3IygR6pYOICdAkJu6ugUsG1hux5vXJH07+ZA897FTJg7oqKyeh03qljEGBl0C2lMWlXZ/6w+FnsFBAjtue9bZpg4Mc74Gs15HkhkQriboEPwQxHazIbqS6nBVfkXk3LsDveqNhEXfW25eRw8ml7sYX7RUNQjlPFQ49zcDCSAHyO/mbqvUvVMcZTjjZ5SABCUUIjEiz2nrL/sbnXuCHLcH0g708V8oVTrZET5boCy72YMzpSlWGgr2VJ8jcW5I20p19AYPW4P6tpIzc= 2021-08-12T02:55:09.166 INFO:teuthology.kill: smithi099.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBjbWAfpLp/BNJkJE+hCSoTJRDtFjv3GU0liszGWCxW+dlHfcgiVUY9OeYUzMDBiZ8EAMjr1Hzpk1P+Is8gIfTOgskfpD2Amk1kfQ62jBciPQJtlUnuv1v+ADFM+LIymM+07LMUShX4Du7bYqsMpivBTiy+BTLgxcyvJSVzm0GpSBgEKa567RGwzL6ik9dTftGmXEJIz+mHrESa1eL26wuwcLBaGPNibrcrfL19gtUrUa7+qRqG2RWGlQ5wV4tTC+wAYbuB60hUf/OZM1W2ct1obYvfArrVe/AyvCJ7m4piFo3leock0UC2MqoPvkYqh2ejjyYVGQYNEqgIB+YLpZoJray5vErQqtIOYWaBzeqAhL55DJwqcQCqK4iExcr9oLQhEms08DLJvanZyY3tgYQRYY4/Nfd0Q0MiPnpsa7fTNIgtZuOxa0AK/Aq5ap4DetENbsYZQAbOe14e9Je3RxeR0eqxV6Dm9dSIp7TZKIAVl0vaiWfGZx69u08/cWv3+U= 2021-08-12T02:55:09.166 INFO:teuthology.kill:2021-08-12 02:55:09,164.164 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-12T02:55:09.167 INFO:teuthology.kill:2021-08-12 02:55:09,167.167 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-12T02:55:09.225 INFO:teuthology.kill:2021-08-12 02:55:09,225.225 INFO:teuthology.orchestra.console:Power off smithi061 2021-08-12T02:55:09.239 INFO:teuthology.kill:2021-08-12 02:55:09,239.239 INFO:teuthology.orchestra.console:Power off smithi099 2021-08-12T02:55:21.705 INFO:teuthology.kill:2021-08-12 02:55:21,705.705 INFO:teuthology.orchestra.console:Power off for smithi061 completed 2021-08-12T02:55:21.710 INFO:teuthology.kill:2021-08-12 02:55:21,710.710 INFO:teuthology.orchestra.console:Power off for smithi099 completed 2021-08-12T02:55:21.932 INFO:teuthology.task.internal:roles: ubuntu@smithi061.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-08-12T02:55:21.933 INFO:teuthology.task.internal:roles: ubuntu@smithi099.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2021-08-12T02:55:21.933 INFO:teuthology.misc:Compressing logs... 2021-08-12T02:55:21.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-12T02:55:56.619 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-08-12T02:55:56.715 INFO:teuthology.kill:No teuthology processes running 2021-08-12T02:55:57.857 INFO:teuthology.kill:Nuking machines: ['smithi061', 'smithi099'] 2021-08-12T02:55:58.805 INFO:teuthology.kill:2021-08-12 02:55:58,805.805 INFO:teuthology.nuke:targets: 2021-08-12T02:55:58.806 INFO:teuthology.kill: smithi061.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq+vxlSRFBYMpDSSSrn1Asq2MQyLtJF4TPx8YTSsK9A4Nw+HeFd/V65RD3iyxnYdC2fm/lnnxDvf3EGh1P2Bp2kuqdo8nzpor9i0yuTfzy5U6aFqkie+tDzUL8e9y+nTbqJEiWGtOqvjwWBQcu8Hf6vOrnqInHDmzbGHDzguRv1aihH3rz1JMVOMZzC7+l34mva76Ow2Lj+/Vn4Unsk/LySUly6kO9Ois9+WUcf8oAMOmjL5jNg3IygR6pYOICdAkJu6ugUsG1hux5vXJH07+ZA897FTJg7oqKyeh03qljEGBl0C2lMWlXZ/6w+FnsFBAjtue9bZpg4Mc74Gs15HkhkQriboEPwQxHazIbqS6nBVfkXk3LsDveqNhEXfW25eRw8ml7sYX7RUNQjlPFQ49zcDCSAHyO/mbqvUvVMcZTjjZ5SABCUUIjEiz2nrL/sbnXuCHLcH0g708V8oVTrZET5boCy72YMzpSlWGgr2VJ8jcW5I20p19AYPW4P6tpIzc= 2021-08-12T02:55:58.807 INFO:teuthology.kill: smithi099.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBjbWAfpLp/BNJkJE+hCSoTJRDtFjv3GU0liszGWCxW+dlHfcgiVUY9OeYUzMDBiZ8EAMjr1Hzpk1P+Is8gIfTOgskfpD2Amk1kfQ62jBciPQJtlUnuv1v+ADFM+LIymM+07LMUShX4Du7bYqsMpivBTiy+BTLgxcyvJSVzm0GpSBgEKa567RGwzL6ik9dTftGmXEJIz+mHrESa1eL26wuwcLBaGPNibrcrfL19gtUrUa7+qRqG2RWGlQ5wV4tTC+wAYbuB60hUf/OZM1W2ct1obYvfArrVe/AyvCJ7m4piFo3leock0UC2MqoPvkYqh2ejjyYVGQYNEqgIB+YLpZoJray5vErQqtIOYWaBzeqAhL55DJwqcQCqK4iExcr9oLQhEms08DLJvanZyY3tgYQRYY4/Nfd0Q0MiPnpsa7fTNIgtZuOxa0AK/Aq5ap4DetENbsYZQAbOe14e9Je3RxeR0eqxV6Dm9dSIp7TZKIAVl0vaiWfGZx69u08/cWv3+U= 2021-08-12T02:55:58.827 INFO:teuthology.kill:2021-08-12 02:55:58,827.827 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-12T02:55:58.830 INFO:teuthology.kill:2021-08-12 02:55:58,830.830 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-12T02:55:58.894 INFO:teuthology.kill:2021-08-12 02:55:58,894.894 INFO:teuthology.orchestra.console:Power off smithi099 2021-08-12T02:55:58.908 INFO:teuthology.kill:2021-08-12 02:55:58,908.908 INFO:teuthology.orchestra.console:Power off smithi061 2021-08-12T02:55:59.038 INFO:teuthology.kill:2021-08-12 02:55:59,038.038 INFO:teuthology.orchestra.console:Power off for smithi099 completed 2021-08-12T02:55:59.051 INFO:teuthology.kill:2021-08-12 02:55:59,051.051 INFO:teuthology.orchestra.console:Power off for smithi061 completed 2021-08-12T02:55:59.184 INFO:teuthology.kill:2021-08-12 02:55:59,183.183 INFO:teuthology.lock.ops:unlocked smithi099.front.sepia.ceph.com 2021-08-12T02:55:59.189 INFO:teuthology.kill:2021-08-12 02:55:59,189.189 INFO:teuthology.lock.ops:unlocked smithi061.front.sepia.ceph.com 2021-08-12T02:57:59.484 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-08-12T02:57:59.579 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-08-12T02:57:59.580 INFO:teuthology.nuke:Checking targets against current locks 2021-08-12T02:57:59.685 INFO:teuthology.nuke:Not nuking smithi061.front.sepia.ceph.com because description doesn't match 2021-08-12T02:57:59.686 INFO:teuthology.nuke:Not nuking smithi099.front.sepia.ceph.com because description doesn't match