2021-09-07T05:08:59.036 INFO:root:teuthology version: 1.1.0-b135a4f8
2021-09-07T05:08:59.133 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi150.front.sepia.ceph.com' reimaging
2021-09-07T05:08:59.134 INFO:teuthology.lock.ops:Updating [ubuntu@smithi150.front.sepia.ceph.com]: reset os type and version on server
2021-09-07T05:08:59.134 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server
2021-09-07T05:08:59.167 INFO:teuthology.lock.ops:Node 'ubuntu@smithi150.front.sepia.ceph.com' reimaging is complete
2021-09-07T05:08:59.167 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi165.front.sepia.ceph.com' reimaging
2021-09-07T05:08:59.168 INFO:teuthology.lock.ops:Updating [ubuntu@smithi165.front.sepia.ceph.com]: reset os type and version on server
2021-09-07T05:08:59.168 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server
2021-09-07T05:08:59.185 INFO:teuthology.lock.ops:Node 'ubuntu@smithi165.front.sepia.ceph.com' reimaging is complete
2021-09-07T05:08:59.390 INFO:teuthology.provision.fog.smithi165:Scheduling deploy of centos 8.2
2021-09-07T05:08:59.397 INFO:teuthology.provision.fog.smithi150:Scheduling deploy of centos 8.2
2021-09-07T05:08:59.785 INFO:teuthology.orchestra.console:Power off smithi165
2021-09-07T05:08:59.821 INFO:teuthology.orchestra.console:Power off smithi150
2021-09-07T05:09:08.182 INFO:teuthology.orchestra.console:Power off for smithi150 completed
2021-09-07T05:09:08.284 INFO:teuthology.orchestra.console:Power on smithi150
2021-09-07T05:09:12.272 INFO:teuthology.orchestra.console:Power off for smithi165 completed
2021-09-07T05:09:12.373 INFO:teuthology.orchestra.console:Power on smithi165
2021-09-07T05:09:16.636 INFO:teuthology.orchestra.console:Power on for smithi165 completed
2021-09-07T05:09:16.704 INFO:teuthology.orchestra.console:Power on for smithi150 completed
2021-09-07T05:09:16.738 INFO:teuthology.provision.fog.smithi165:Waiting for deploy to finish
2021-09-07T05:09:16.805 INFO:teuthology.provision.fog.smithi150:Waiting for deploy to finish
2021-09-07T05:13:41.120 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-09-07T05:13:56.450 INFO:teuthology.provision.fog.smithi165:Node is ready
2021-09-07T05:13:56.470 INFO:teuthology.orchestra.run.smithi165.stdout:smithi165.front.sepia.ceph.com
2021-09-07T05:13:56.526 INFO:teuthology.orchestra.run.smithi165.stdout:172.21.15.165 smithi165.front.sepia.ceph.com smithi165
2021-09-07T05:13:56.876 INFO:teuthology.provision.fog.smithi165:Deploy complete!
2021-09-07T05:13:58.090 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-09-07T05:13:58.399 INFO:teuthology.provision.fog.smithi150:Node is ready
2021-09-07T05:13:58.413 INFO:teuthology.orchestra.run.smithi150.stdout:smithi150.front.sepia.ceph.com
2021-09-07T05:13:58.469 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150 smithi150.front.sepia.ceph.com smithi150
2021-09-07T05:13:58.795 INFO:teuthology.provision.fog.smithi150:Deploy complete!
2021-09-07T05:13:58.931 INFO:teuthology.lock.ops:Checking smithi150.front.sepia.ceph.com
2021-09-07T05:13:58.932 INFO:teuthology.lock.ops:New key found. Updating...
2021-09-07T05:13:58.967 INFO:teuthology.lock.ops:Checking smithi165.front.sepia.ceph.com
2021-09-07T05:13:58.967 INFO:teuthology.lock.ops:New key found. Updating...
2021-09-07T05:13:58.985 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: set os type and version on server
2021-09-07T05:13:58.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-09-07T05:13:59.242 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64
2021-09-07T05:13:59.298 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Linux"
2021-09-07T05:13:59.298 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="8 (Core)"
2021-09-07T05:13:59.299 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos"
2021-09-07T05:13:59.299 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora"
2021-09-07T05:13:59.300 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="8"
2021-09-07T05:13:59.300 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el8"
2021-09-07T05:13:59.300 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Linux 8 (Core)"
2021-09-07T05:13:59.301 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31"
2021-09-07T05:13:59.302 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-09-07T05:13:59.302 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://www.centos.org/"
2021-09-07T05:13:59.303 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-09-07T05:13:59.304 INFO:teuthology.orchestra.run.smithi150.stdout:
2021-09-07T05:13:59.304 INFO:teuthology.orchestra.run.smithi150.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-09-07T05:13:59.304 INFO:teuthology.orchestra.run.smithi150.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-09-07T05:13:59.305 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="centos"
2021-09-07T05:13:59.305 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8"
2021-09-07T05:13:59.306 INFO:teuthology.orchestra.run.smithi150.stdout:
2021-09-07T05:13:59.308 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server
2021-09-07T05:13:59.330 INFO:teuthology.lock.ops:Updating [smithi165.front.sepia.ceph.com]: set os type and version on server
2021-09-07T05:13:59.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-09-07T05:13:59.542 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64
2021-09-07T05:13:59.597 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="CentOS Linux"
2021-09-07T05:13:59.597 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="8 (Core)"
2021-09-07T05:13:59.598 INFO:teuthology.orchestra.run.smithi165.stdout:ID="centos"
2021-09-07T05:13:59.598 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE="rhel fedora"
2021-09-07T05:13:59.598 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="8"
2021-09-07T05:13:59.598 INFO:teuthology.orchestra.run.smithi165.stdout:PLATFORM_ID="platform:el8"
2021-09-07T05:13:59.599 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="CentOS Linux 8 (Core)"
2021-09-07T05:13:59.599 INFO:teuthology.orchestra.run.smithi165.stdout:ANSI_COLOR="0;31"
2021-09-07T05:13:59.599 INFO:teuthology.orchestra.run.smithi165.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-09-07T05:13:59.599 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://www.centos.org/"
2021-09-07T05:13:59.599 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-09-07T05:13:59.600 INFO:teuthology.orchestra.run.smithi165.stdout:
2021-09-07T05:13:59.600 INFO:teuthology.orchestra.run.smithi165.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-09-07T05:13:59.600 INFO:teuthology.orchestra.run.smithi165.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-09-07T05:13:59.600 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT="centos"
2021-09-07T05:13:59.601 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8"
2021-09-07T05:13:59.601 INFO:teuthology.orchestra.run.smithi165.stdout:
2021-09-07T05:13:59.602 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server
2021-09-07T05:13:59.696 INFO:teuthology.dispatcher.supervisor:Running job 6377539
2021-09-07T05:13:59.696 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/virtualenv/bin/teuthology -v --owner scheduled_adking@teuthology --archive /home/teuthworker/archive/adking-2021-09-07_04:51:05-rados:cephadm-wip-adk-testing-2021-09-06-1946-distro-basic-smithi/6377539 --name adking-2021-09-07_04:51:05-rados:cephadm-wip-adk-testing-2021-09-06-1946-distro-basic-smithi --description rados:cephadm/mgr-nfs-upgrade/{0-centos_8.2_container_tools_3.0 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/adking-2021-09-07_04:51:05-rados:cephadm-wip-adk-testing-2021-09-06-1946-distro-basic-smithi/6377539/orig.config.yaml
2021-09-07T05:13:59.700 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/adking-2021-09-07_04:51:05-rados:cephadm-wip-adk-testing-2021-09-06-1946-distro-basic-smithi/6377539
2021-09-07T05:13:59.701 INFO:teuthology.dispatcher.supervisor:Job PID: 482
2021-09-07T05:13:59.701 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-09-07T05:38:01.690 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/adking-2021-09-07_04:51:05-rados:cephadm-wip-adk-testing-2021-09-06-1946-distro-basic-smithi/jobs/ failed with status 500:
Internal Server Error
Internal Server Error
2021-09-07T17:14:19.825 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-09-07T17:14:20.039 INFO:teuthology.kill:Killing Pids: {482}
2021-09-07T17:14:21.090 INFO:teuthology.kill:Nuking machines: ['smithi150', 'smithi165']
2021-09-07T17:14:22.039 INFO:teuthology.kill:2021-09-07 17:14:22,039.039 INFO:teuthology.nuke:targets:
2021-09-07T17:14:22.040 INFO:teuthology.kill: smithi150.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDoiwcQtL18+pbxvOcD7MybOHyPcyy/ySHq6ZfsDJLO9FdgOFvMXeAq1/eeQvpvO5vmAlXQZ8dhfXONlAgKY5y+73mZ+NdnHzknpX1/fZOJ8Mor+kU6Td6ZQgv6RA1kNyF8yWk8WG6mJGo7Q/BdoSAYFxJ6hXTUwVb6f0NgnhmBU6WKdo2/AXPWDc5IacIWf87RwDcDaaBAahkgXseJ2OZ6xojXYCruIBUc2lJfEXcNtLlV9vjSoSbgEqAmRFJkdxto/1hAOY9gQ4qBRHwMMsC1MfeRNqopV5nL0V/3GFT50HDoVIbokjf31N/Wd34V1/HCfxYCoD60FwSxClcI0upu9ZFVQ0DOZq9okQ/VSLFe0nQvv2F3gZJE9PkHlQRAST0oVIJUK+JLLJL6aKvT2IkbamUFL1UXyaJ6SRCGl6dOtgis51wpak5ZJ0eurhXy5qlNjlqDgnXjS/+sB96SQKp4w9Ofbofb97uDTb1Fm65ot/rpCkBngor28SLvCO9hl40=
2021-09-07T17:14:22.040 INFO:teuthology.kill: smithi165.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnYtYhZMgKttX1OvgI3qHCmY7HehBYo8P2qrqz7LSUJ3ZcKkePbCwsQzMgHkTZUtda2ZDIpQKYw01RFVIuykHVWegqFGQXpeiAL0wcJdY+qaL+55kfehy1dZKi35injHYFWK8jF6UctHozafY1nxMuXHWOeJe5KQegO0ai0NkUOWOIZl5XMysMwxAwJNeNtTZd8BZL+xuSWXChV9pTUax7Hiz4G9tq22NNuflRq7m8FPwZyynT0KgpP64c0J0WsqCCTXRW7/uSQeCrQZmf86wlk2z/D3R/D/pASGsPLUGA049SzVEk0NPejattrVaKF2y0xYMcnt5q3Ok0ZF+RkTNlTwH9cAVdygrrfWbLNOAKPgEI4j/vA6DaqoNCflpTlYAnhqCtW6mXH52+Xnufue2s1lJUWUSePrPTLS/xWxflpLXSMKQz94kFF8gaCijWEMJKifVzVhURBDQtfHPJ54eg2jKBF9xcxhJC2Rqu3vQVc+Q4iHiHqGwQJvI6DFqF5Ak=
2021-09-07T17:14:22.040 INFO:teuthology.kill:2021-09-07 17:14:22,039.039 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-09-07T17:14:22.042 INFO:teuthology.kill:2021-09-07 17:14:22,042.042 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-09-07T17:14:22.100 INFO:teuthology.kill:2021-09-07 17:14:22,100.100 INFO:teuthology.orchestra.console:Power off smithi150
2021-09-07T17:14:22.112 INFO:teuthology.kill:2021-09-07 17:14:22,112.112 INFO:teuthology.orchestra.console:Power off smithi165
2021-09-07T17:14:30.465 INFO:teuthology.kill:2021-09-07 17:14:30,465.465 INFO:teuthology.orchestra.console:Power off for smithi150 completed
2021-09-07T17:14:30.472 INFO:teuthology.kill:2021-09-07 17:14:30,471.471 INFO:teuthology.orchestra.console:Power off for smithi165 completed
2021-09-07T17:14:30.704 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2021-09-07T17:14:30.705 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2021-09-07T17:14:30.705 INFO:teuthology.misc:Compressing logs...
2021-09-07T17:14:30.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-09-07T17:15:30.770 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
teuth_config.archive_base, job_config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
compress_logs(ctx, log_path)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/misc.py", line 1378, in compress_logs
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/cluster.py", line 64, in
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/remote.py", line 509, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/run.py", line 434, in run
transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-09-07T17:15:30.825 INFO:teuthology.kill:No teuthology processes running
2021-09-07T17:15:31.894 INFO:teuthology.kill:Nuking machines: ['smithi150', 'smithi165']
2021-09-07T17:15:32.815 INFO:teuthology.kill:2021-09-07 17:15:32,815.815 INFO:teuthology.nuke:targets:
2021-09-07T17:15:32.816 INFO:teuthology.kill: smithi150.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDoiwcQtL18+pbxvOcD7MybOHyPcyy/ySHq6ZfsDJLO9FdgOFvMXeAq1/eeQvpvO5vmAlXQZ8dhfXONlAgKY5y+73mZ+NdnHzknpX1/fZOJ8Mor+kU6Td6ZQgv6RA1kNyF8yWk8WG6mJGo7Q/BdoSAYFxJ6hXTUwVb6f0NgnhmBU6WKdo2/AXPWDc5IacIWf87RwDcDaaBAahkgXseJ2OZ6xojXYCruIBUc2lJfEXcNtLlV9vjSoSbgEqAmRFJkdxto/1hAOY9gQ4qBRHwMMsC1MfeRNqopV5nL0V/3GFT50HDoVIbokjf31N/Wd34V1/HCfxYCoD60FwSxClcI0upu9ZFVQ0DOZq9okQ/VSLFe0nQvv2F3gZJE9PkHlQRAST0oVIJUK+JLLJL6aKvT2IkbamUFL1UXyaJ6SRCGl6dOtgis51wpak5ZJ0eurhXy5qlNjlqDgnXjS/+sB96SQKp4w9Ofbofb97uDTb1Fm65ot/rpCkBngor28SLvCO9hl40=
2021-09-07T17:15:32.816 INFO:teuthology.kill: smithi165.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnYtYhZMgKttX1OvgI3qHCmY7HehBYo8P2qrqz7LSUJ3ZcKkePbCwsQzMgHkTZUtda2ZDIpQKYw01RFVIuykHVWegqFGQXpeiAL0wcJdY+qaL+55kfehy1dZKi35injHYFWK8jF6UctHozafY1nxMuXHWOeJe5KQegO0ai0NkUOWOIZl5XMysMwxAwJNeNtTZd8BZL+xuSWXChV9pTUax7Hiz4G9tq22NNuflRq7m8FPwZyynT0KgpP64c0J0WsqCCTXRW7/uSQeCrQZmf86wlk2z/D3R/D/pASGsPLUGA049SzVEk0NPejattrVaKF2y0xYMcnt5q3Ok0ZF+RkTNlTwH9cAVdygrrfWbLNOAKPgEI4j/vA6DaqoNCflpTlYAnhqCtW6mXH52+Xnufue2s1lJUWUSePrPTLS/xWxflpLXSMKQz94kFF8gaCijWEMJKifVzVhURBDQtfHPJ54eg2jKBF9xcxhJC2Rqu3vQVc+Q4iHiHqGwQJvI6DFqF5Ak=
2021-09-07T17:15:32.840 INFO:teuthology.kill:2021-09-07 17:15:32,840.840 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-09-07T17:15:32.843 INFO:teuthology.kill:2021-09-07 17:15:32,843.843 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-09-07T17:15:32.900 INFO:teuthology.kill:2021-09-07 17:15:32,900.900 INFO:teuthology.orchestra.console:Power off smithi165
2021-09-07T17:15:32.913 INFO:teuthology.kill:2021-09-07 17:15:32,913.913 INFO:teuthology.orchestra.console:Power off smithi150
2021-09-07T17:15:33.039 INFO:teuthology.kill:2021-09-07 17:15:33,038.038 INFO:teuthology.orchestra.console:Power off for smithi165 completed
2021-09-07T17:15:33.050 INFO:teuthology.kill:2021-09-07 17:15:33,050.050 INFO:teuthology.orchestra.console:Power off for smithi150 completed
2021-09-07T17:15:33.191 INFO:teuthology.kill:2021-09-07 17:15:33,190.190 INFO:teuthology.lock.ops:unlocked smithi165.front.sepia.ceph.com
2021-09-07T17:15:33.193 INFO:teuthology.kill:2021-09-07 17:15:33,193.193 INFO:teuthology.lock.ops:unlocked smithi150.front.sepia.ceph.com
2021-09-07T17:17:33.387 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15