2021-02-24T17:58:00.178 INFO:root:teuthology version: 1.1.0-9a6b849c 2021-02-24T17:58:00.282 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi031.front.sepia.ceph.com' reimaging 2021-02-24T17:58:00.283 INFO:teuthology.lock.ops:Updating [ubuntu@smithi031.front.sepia.ceph.com]: reset os type and version on server 2021-02-24T17:58:00.284 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2021-02-24T17:58:00.303 INFO:teuthology.lock.ops:Node 'ubuntu@smithi031.front.sepia.ceph.com' reimaging is complete 2021-02-24T17:58:00.303 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi053.front.sepia.ceph.com' reimaging 2021-02-24T17:58:00.303 INFO:teuthology.lock.ops:Updating [ubuntu@smithi053.front.sepia.ceph.com]: reset os type and version on server 2021-02-24T17:58:00.304 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2021-02-24T17:58:00.317 INFO:teuthology.lock.ops:Node 'ubuntu@smithi053.front.sepia.ceph.com' reimaging is complete 2021-02-24T17:58:00.526 INFO:teuthology.provision.fog.smithi053:Scheduling deploy of centos 8.2 2021-02-24T17:58:00.538 INFO:teuthology.provision.fog.smithi031:Scheduling deploy of centos 8.2 2021-02-24T17:58:00.898 INFO:teuthology.orchestra.console:Power off smithi031 2021-02-24T17:58:00.936 INFO:teuthology.orchestra.console:Power off smithi053 2021-02-24T17:58:01.048 INFO:teuthology.orchestra.console:Power off for smithi031 completed 2021-02-24T17:58:01.084 INFO:teuthology.orchestra.console:Power off for smithi053 completed 2021-02-24T17:58:01.150 INFO:teuthology.orchestra.console:Power on smithi031 2021-02-24T17:58:01.188 INFO:teuthology.orchestra.console:Power on smithi053 2021-02-24T17:58:05.422 INFO:teuthology.orchestra.console:Power on for smithi031 completed 2021-02-24T17:58:05.452 INFO:teuthology.orchestra.console:Power on for smithi053 completed 2021-02-24T17:58:05.523 INFO:teuthology.provision.fog.smithi031:Waiting for deploy to finish 2021-02-24T17:58:05.554 INFO:teuthology.provision.fog.smithi053:Waiting for deploy to finish 2021-02-24T18:02:22.252 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-24T18:02:22.575 INFO:teuthology.provision.fog.smithi031:Node is ready 2021-02-24T18:02:22.590 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031.front.sepia.ceph.com 2021-02-24T18:02:22.646 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31 smithi031.front.sepia.ceph.com smithi031 2021-02-24T18:02:22.981 INFO:teuthology.provision.fog.smithi031:Deploy complete! 2021-02-24T18:04:02.072 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-24T18:04:02.398 INFO:teuthology.provision.fog.smithi053:Node is ready 2021-02-24T18:04:02.413 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053.front.sepia.ceph.com 2021-02-24T18:04:02.470 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53 smithi053.front.sepia.ceph.com smithi053 2021-02-24T18:04:02.912 INFO:teuthology.provision.fog.smithi053:Deploy complete! 2021-02-24T18:04:03.061 INFO:teuthology.lock.ops:Checking smithi031.front.sepia.ceph.com 2021-02-24T18:04:03.062 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-24T18:04:03.076 INFO:teuthology.lock.ops:Checking smithi053.front.sepia.ceph.com 2021-02-24T18:04:03.077 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-24T18:04:03.093 INFO:teuthology.lock.ops:Updating [smithi031.front.sepia.ceph.com]: set os type and version on server 2021-02-24T18:04:03.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-24T18:04:03.374 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2021-02-24T18:04:03.429 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Linux" 2021-02-24T18:04:03.429 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="8 (Core)" 2021-02-24T18:04:03.430 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2021-02-24T18:04:03.430 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2021-02-24T18:04:03.430 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="8" 2021-02-24T18:04:03.430 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el8" 2021-02-24T18:04:03.431 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-02-24T18:04:03.431 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2021-02-24T18:04:03.431 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-02-24T18:04:03.431 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://www.centos.org/" 2021-02-24T18:04:03.432 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-02-24T18:04:03.432 INFO:teuthology.orchestra.run.smithi031.stdout: 2021-02-24T18:04:03.432 INFO:teuthology.orchestra.run.smithi031.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-02-24T18:04:03.432 INFO:teuthology.orchestra.run.smithi031.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-02-24T18:04:03.432 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-02-24T18:04:03.433 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-02-24T18:04:03.433 INFO:teuthology.orchestra.run.smithi031.stdout: 2021-02-24T18:04:03.434 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2021-02-24T18:04:03.451 INFO:teuthology.lock.ops:Updating [smithi053.front.sepia.ceph.com]: set os type and version on server 2021-02-24T18:04:03.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-24T18:04:03.773 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2021-02-24T18:04:03.830 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Linux" 2021-02-24T18:04:03.831 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="8 (Core)" 2021-02-24T18:04:03.832 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2021-02-24T18:04:03.832 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2021-02-24T18:04:03.833 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="8" 2021-02-24T18:04:03.833 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el8" 2021-02-24T18:04:03.834 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-02-24T18:04:03.834 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2021-02-24T18:04:03.834 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-02-24T18:04:03.835 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://www.centos.org/" 2021-02-24T18:04:03.835 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-02-24T18:04:03.836 INFO:teuthology.orchestra.run.smithi053.stdout: 2021-02-24T18:04:03.836 INFO:teuthology.orchestra.run.smithi053.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-02-24T18:04:03.837 INFO:teuthology.orchestra.run.smithi053.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-02-24T18:04:03.837 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-02-24T18:04:03.838 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-02-24T18:04:03.839 INFO:teuthology.orchestra.run.smithi053.stdout: 2021-02-24T18:04:03.840 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2021-02-24T18:04:03.931 INFO:teuthology.dispatcher.supervisor:Running job 5910533 2021-02-24T18:04:03.932 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-02-24_03:31:03-rados-pacific-distro-basic-smithi/5910533 --name teuthology-2021-02-24_03:31:03-rados-pacific-distro-basic-smithi --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/valgrind} -- /home/teuthworker/archive/teuthology-2021-02-24_03:31:03-rados-pacific-distro-basic-smithi/5910533/orig.config.yaml 2021-02-24T18:04:03.938 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-02-24_03:31:03-rados-pacific-distro-basic-smithi/5910533 2021-02-24T18:04:03.939 INFO:teuthology.dispatcher.supervisor:Job PID: 31402 2021-02-24T18:04:03.939 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-02-25T06:04:23.078 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-02-25T06:04:23.329 INFO:teuthology.kill:Killing Pids: {31402} 2021-02-25T06:04:24.309 INFO:teuthology.kill:Nuking machines: ['smithi031', 'smithi053'] 2021-02-25T06:04:25.131 INFO:teuthology.kill:2021-02-25 06:04:25,131.131 INFO:teuthology.nuke:targets: 2021-02-25T06:04:25.132 INFO:teuthology.kill: smithi031.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQqY6pchVxtzdPOg8ZG7ohcL3EQ8L9p7DSgIi8wjOoSnK8LsDuODZDlT52cZuVamKeS/17fk323sUwEhysQ0L7riUprcQDhgWvcrtsbxbDUyeRkgh9zlBaxyZy0UfTAzPkOuOTnwiqpudzYaofebmlupwhMT9cr4qPiMg/ZL9S0cifNBIfs5klz1djPkG0MxYr3XFzj5QImzpbo0EazDNtMZ3evPsKHy3BZZdhhA3bDOpf1Bto5onGFECF0AR9MqVeNgFKRor8eFTnYKei7B+Du5FYXTFuaShCvEWP80tLRyuJwQThuhCoam8oueT3V1/Ed13I3TzYiiSlcCcHI4H0FPWBLRHLAx/YjYTNQihfVyKLRoQox9NsaQFzQ4yTmzb83AjrgnTKBehFbFB6tjj9h43/sDV+pdiHW91dzs/fhzf2hZJbQ7wiBGwQ7J/PKoiCAg/0bdSfUynBiA1NgRh8paMCs3pD073ovocto3EemRnOElyPQ7+bor1cwXgj8GU= 2021-02-25T06:04:25.132 INFO:teuthology.kill: smithi053.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvlzTKYtP0H2sOeLO01u3HAi3ibDzRYdh9saptqCCEWDE/tJYML0sI/VxCZZAK7ig+Na+iJx56RAOxceA9C2qgtAhuYp5tKOhv+r/CgILr70BKxhwp3SCBzIhlt/6eUjBpziic0dIUyiJS7rQk3J5FDlbEAaJjAfBlok4rAsZ9cjXduWXEdX71QQEBRSH0c2BFJpt70S8S6yZoA65maS7OQVR673KS4drAeU+7b+OVddJRR3PFJEMKfefz6F0jS/abjaXvnHZokcExjWu5T4YNWB2Iq4kfdEmLWnQFn51mWgjPLWbzivqwJan1W35kjHGGX0bWZz/rDnRTnFhA0ror0+4v22xQX0FYS0V5oPqpXEaTLV3Me3yKYKOeByOAL4H8mIJyeWqM8UiV7vK34PaQz0pLzzY9j2p0Kl4gUwSEh8yC7mhy44EkBZqo+OXLOXbvtNMs7IV+tfhyvGIe+xOvzrYuJD3n7RAG8JHnuxl9ON4BmV3ArQWRmd9hC2ccv1s= 2021-02-25T06:04:25.133 INFO:teuthology.kill:2021-02-25 06:04:25,131.131 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-25T06:04:25.138 INFO:teuthology.kill:2021-02-25 06:04:25,137.137 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-25T06:04:25.203 INFO:teuthology.kill:2021-02-25 06:04:25,202.202 INFO:teuthology.orchestra.console:Power off smithi031 2021-02-25T06:04:25.230 INFO:teuthology.kill:2021-02-25 06:04:25,229.229 INFO:teuthology.orchestra.console:Power off smithi053 2021-02-25T06:04:33.598 INFO:teuthology.kill:2021-02-25 06:04:33,597.597 INFO:teuthology.orchestra.console:Power off for smithi031 completed 2021-02-25T06:04:33.609 INFO:teuthology.kill:2021-02-25 06:04:33,609.609 INFO:teuthology.orchestra.console:Power off for smithi053 completed 2021-02-25T06:04:33.880 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-02-25T06:04:33.881 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-02-25T06:04:33.881 INFO:teuthology.misc:Compressing logs... 2021-02-25T06:04:33.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-25T06:05:08.423 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-02-25T06:05:08.537 INFO:teuthology.kill:No teuthology processes running 2021-02-25T06:05:09.767 INFO:teuthology.kill:Nuking machines: ['smithi031', 'smithi053'] 2021-02-25T06:05:10.639 INFO:teuthology.kill:2021-02-25 06:05:10,639.639 INFO:teuthology.nuke:targets: 2021-02-25T06:05:10.640 INFO:teuthology.kill: smithi031.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQqY6pchVxtzdPOg8ZG7ohcL3EQ8L9p7DSgIi8wjOoSnK8LsDuODZDlT52cZuVamKeS/17fk323sUwEhysQ0L7riUprcQDhgWvcrtsbxbDUyeRkgh9zlBaxyZy0UfTAzPkOuOTnwiqpudzYaofebmlupwhMT9cr4qPiMg/ZL9S0cifNBIfs5klz1djPkG0MxYr3XFzj5QImzpbo0EazDNtMZ3evPsKHy3BZZdhhA3bDOpf1Bto5onGFECF0AR9MqVeNgFKRor8eFTnYKei7B+Du5FYXTFuaShCvEWP80tLRyuJwQThuhCoam8oueT3V1/Ed13I3TzYiiSlcCcHI4H0FPWBLRHLAx/YjYTNQihfVyKLRoQox9NsaQFzQ4yTmzb83AjrgnTKBehFbFB6tjj9h43/sDV+pdiHW91dzs/fhzf2hZJbQ7wiBGwQ7J/PKoiCAg/0bdSfUynBiA1NgRh8paMCs3pD073ovocto3EemRnOElyPQ7+bor1cwXgj8GU= 2021-02-25T06:05:10.640 INFO:teuthology.kill: smithi053.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvlzTKYtP0H2sOeLO01u3HAi3ibDzRYdh9saptqCCEWDE/tJYML0sI/VxCZZAK7ig+Na+iJx56RAOxceA9C2qgtAhuYp5tKOhv+r/CgILr70BKxhwp3SCBzIhlt/6eUjBpziic0dIUyiJS7rQk3J5FDlbEAaJjAfBlok4rAsZ9cjXduWXEdX71QQEBRSH0c2BFJpt70S8S6yZoA65maS7OQVR673KS4drAeU+7b+OVddJRR3PFJEMKfefz6F0jS/abjaXvnHZokcExjWu5T4YNWB2Iq4kfdEmLWnQFn51mWgjPLWbzivqwJan1W35kjHGGX0bWZz/rDnRTnFhA0ror0+4v22xQX0FYS0V5oPqpXEaTLV3Me3yKYKOeByOAL4H8mIJyeWqM8UiV7vK34PaQz0pLzzY9j2p0Kl4gUwSEh8yC7mhy44EkBZqo+OXLOXbvtNMs7IV+tfhyvGIe+xOvzrYuJD3n7RAG8JHnuxl9ON4BmV3ArQWRmd9hC2ccv1s= 2021-02-25T06:05:10.666 INFO:teuthology.kill:2021-02-25 06:05:10,665.665 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-25T06:05:10.668 INFO:teuthology.kill:2021-02-25 06:05:10,668.668 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-25T06:05:10.728 INFO:teuthology.kill:2021-02-25 06:05:10,728.728 INFO:teuthology.orchestra.console:Power off smithi031 2021-02-25T06:05:10.744 INFO:teuthology.kill:2021-02-25 06:05:10,743.743 INFO:teuthology.orchestra.console:Power off smithi053 2021-02-25T06:05:10.879 INFO:teuthology.kill:2021-02-25 06:05:10,878.878 INFO:teuthology.orchestra.console:Power off for smithi031 completed 2021-02-25T06:05:10.893 INFO:teuthology.kill:2021-02-25 06:05:10,893.893 INFO:teuthology.orchestra.console:Power off for smithi053 completed 2021-02-25T06:05:11.021 INFO:teuthology.kill:2021-02-25 06:05:11,020.020 INFO:teuthology.lock.ops:unlocked smithi031.front.sepia.ceph.com 2021-02-25T06:05:11.033 INFO:teuthology.kill:2021-02-25 06:05:11,033.033 INFO:teuthology.lock.ops:unlocked smithi053.front.sepia.ceph.com 2021-02-25T06:07:11.268 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-02-25T06:07:11.391 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-02-25T06:07:11.392 INFO:teuthology.nuke:Checking targets against current locks 2021-02-25T06:07:11.481 INFO:teuthology.nuke:Not nuking smithi053.front.sepia.ceph.com because description doesn't match 2021-02-25T06:07:11.482 INFO:teuthology.nuke:Not nuking smithi031.front.sepia.ceph.com because description doesn't match