2021-03-14T13:36:10.949 INFO:root:teuthology version: 1.1.0-b9656917 2021-03-14T13:36:11.060 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi061.front.sepia.ceph.com' reimaging 2021-03-14T13:36:11.061 INFO:teuthology.lock.ops:Updating [ubuntu@smithi061.front.sepia.ceph.com]: reset os type and version on server 2021-03-14T13:36:11.061 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server 2021-03-14T13:36:11.079 INFO:teuthology.lock.ops:Node 'ubuntu@smithi061.front.sepia.ceph.com' reimaging is complete 2021-03-14T13:36:11.079 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi084.front.sepia.ceph.com' reimaging 2021-03-14T13:36:11.080 INFO:teuthology.lock.ops:Updating [ubuntu@smithi084.front.sepia.ceph.com]: reset os type and version on server 2021-03-14T13:36:11.080 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2021-03-14T13:36:11.097 INFO:teuthology.lock.ops:Node 'ubuntu@smithi084.front.sepia.ceph.com' reimaging is complete 2021-03-14T13:36:11.290 INFO:teuthology.provision.fog.smithi061:Scheduling deploy of centos 8.3 2021-03-14T13:36:11.295 INFO:teuthology.provision.fog.smithi084:Scheduling deploy of centos 8.3 2021-03-14T13:36:11.797 INFO:teuthology.orchestra.console:Power off smithi061 2021-03-14T13:36:11.840 INFO:teuthology.orchestra.console:Power off smithi084 2021-03-14T13:36:20.222 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2021-03-14T13:36:20.323 INFO:teuthology.orchestra.console:Power on smithi084 2021-03-14T13:36:24.331 INFO:teuthology.orchestra.console:Power off for smithi061 completed 2021-03-14T13:36:24.433 INFO:teuthology.orchestra.console:Power on smithi061 2021-03-14T13:36:28.696 INFO:teuthology.orchestra.console:Power on for smithi084 completed 2021-03-14T13:36:28.797 INFO:teuthology.provision.fog.smithi084:Waiting for deploy to finish 2021-03-14T13:36:32.801 INFO:teuthology.orchestra.console:Power on for smithi061 completed 2021-03-14T13:36:32.902 INFO:teuthology.provision.fog.smithi061:Waiting for deploy to finish 2021-03-14T13:39:00.435 ERROR:teuthology.orchestra.connection:Error authenticating with smithi084.front.sepia.ceph.com: Authentication failed. 2021-03-14T13:39:01.534 ERROR:teuthology.orchestra.connection:Error authenticating with smithi084.front.sepia.ceph.com: Authentication failed. 2021-03-14T13:39:02.632 ERROR:teuthology.orchestra.connection:Error authenticating with smithi084.front.sepia.ceph.com: Authentication failed. 2021-03-14T13:40:43.284 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-14T13:41:03.643 INFO:teuthology.provision.fog.smithi084:Node is ready 2021-03-14T13:41:03.663 INFO:teuthology.orchestra.run.smithi084.stdout:smithi084.front.sepia.ceph.com 2021-03-14T13:41:03.720 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84 smithi084.front.sepia.ceph.com smithi084 2021-03-14T13:41:04.072 INFO:teuthology.provision.fog.smithi084:Deploy complete! 2021-03-14T13:41:59.330 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-14T13:41:59.662 INFO:teuthology.provision.fog.smithi061:Node is ready 2021-03-14T13:41:59.677 INFO:teuthology.orchestra.run.smithi061.stdout:smithi061.front.sepia.ceph.com 2021-03-14T13:41:59.732 INFO:teuthology.orchestra.run.smithi061.stdout:172.21.15.61 smithi061.front.sepia.ceph.com smithi061 2021-03-14T13:42:00.061 INFO:teuthology.provision.fog.smithi061:Deploy complete! 2021-03-14T13:42:00.182 INFO:teuthology.lock.ops:Checking smithi061.front.sepia.ceph.com 2021-03-14T13:42:00.183 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-14T13:42:00.199 INFO:teuthology.lock.ops:Checking smithi084.front.sepia.ceph.com 2021-03-14T13:42:00.200 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-14T13:42:00.215 INFO:teuthology.lock.ops:Updating [smithi061.front.sepia.ceph.com]: set os type and version on server 2021-03-14T13:42:00.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-14T13:42:00.502 INFO:teuthology.orchestra.run.smithi061.stdout:x86_64 2021-03-14T13:42:00.556 INFO:teuthology.orchestra.run.smithi061.stdout:NAME="CentOS Linux" 2021-03-14T13:42:00.557 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION="8" 2021-03-14T13:42:00.557 INFO:teuthology.orchestra.run.smithi061.stdout:ID="centos" 2021-03-14T13:42:00.557 INFO:teuthology.orchestra.run.smithi061.stdout:ID_LIKE="rhel fedora" 2021-03-14T13:42:00.558 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_ID="8" 2021-03-14T13:42:00.558 INFO:teuthology.orchestra.run.smithi061.stdout:PLATFORM_ID="platform:el8" 2021-03-14T13:42:00.558 INFO:teuthology.orchestra.run.smithi061.stdout:PRETTY_NAME="CentOS Linux 8" 2021-03-14T13:42:00.558 INFO:teuthology.orchestra.run.smithi061.stdout:ANSI_COLOR="0;31" 2021-03-14T13:42:00.559 INFO:teuthology.orchestra.run.smithi061.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-03-14T13:42:00.559 INFO:teuthology.orchestra.run.smithi061.stdout:HOME_URL="https://centos.org/" 2021-03-14T13:42:00.559 INFO:teuthology.orchestra.run.smithi061.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-03-14T13:42:00.559 INFO:teuthology.orchestra.run.smithi061.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-03-14T13:42:00.559 INFO:teuthology.orchestra.run.smithi061.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-03-14T13:42:00.561 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server 2021-03-14T13:42:00.575 INFO:teuthology.lock.ops:Updating [smithi084.front.sepia.ceph.com]: set os type and version on server 2021-03-14T13:42:00.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-14T13:42:00.791 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2021-03-14T13:42:00.846 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="CentOS Linux" 2021-03-14T13:42:00.847 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="8" 2021-03-14T13:42:00.847 INFO:teuthology.orchestra.run.smithi084.stdout:ID="centos" 2021-03-14T13:42:00.847 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="rhel fedora" 2021-03-14T13:42:00.847 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="8" 2021-03-14T13:42:00.848 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el8" 2021-03-14T13:42:00.848 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="CentOS Linux 8" 2021-03-14T13:42:00.848 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2021-03-14T13:42:00.848 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-03-14T13:42:00.849 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://centos.org/" 2021-03-14T13:42:00.849 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-03-14T13:42:00.849 INFO:teuthology.orchestra.run.smithi084.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-03-14T13:42:00.849 INFO:teuthology.orchestra.run.smithi084.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-03-14T13:42:00.850 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2021-03-14T13:42:00.944 INFO:teuthology.dispatcher.supervisor:Running job 5964986 2021-03-14T13:42:00.945 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/bin/teuthology -v --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2021-03-14_09:40:23-rados-wip-kefu-testing-2021-03-14-1612-distro-basic-smithi/5964986 --name kchai-2021-03-14_09:40:23-rados-wip-kefu-testing-2021-03-14-1612-distro-basic-smithi --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/valgrind} -- /home/teuthworker/archive/kchai-2021-03-14_09:40:23-rados-wip-kefu-testing-2021-03-14-1612-distro-basic-smithi/5964986/orig.config.yaml 2021-03-14T13:42:00.949 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/kchai-2021-03-14_09:40:23-rados-wip-kefu-testing-2021-03-14-1612-distro-basic-smithi/5964986 2021-03-14T13:42:00.950 INFO:teuthology.dispatcher.supervisor:Job PID: 652 2021-03-14T13:42:00.950 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-03-15T01:42:16.421 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-15T01:42:16.643 INFO:teuthology.kill:Killing Pids: {652} 2021-03-15T01:42:18.548 INFO:teuthology.kill:Nuking machines: ['smithi061', 'smithi084'] 2021-03-15T01:42:20.350 INFO:teuthology.kill:2021-03-15 01:42:20,350.350 INFO:teuthology.nuke:targets: 2021-03-15T01:42:20.354 INFO:teuthology.kill: smithi061.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVSC0xCfgsUYukMwgp3J4jX5sD7v3YZOiJ8x5MBX5UCPcUf1sXviNfnRTRx+ZhM5vPurglLGK111n2IXeWAtlYJnWTbLTW/beU44gen5lZE6aX4Ljk1eZBGfvUwDY7H2XRZse9OZghzVQtQHQhDhTwPUqb+7BmrfGA9z6CPnq+ycECdf9WAcB4nvKt+BJzd1v2CPLx3csqKH2YK1GvT0Vxr+CDG4YkitrCqe2LgHM99rQAh1Jb80FRIW1tTAETWDV68qxT5uXyvBTUCkM/qLtuFr8/Sh2DfMXKbIMkc+7P5zyFcp/9efTEkzPxvnVNGRJDulfiKE6w/z+/G5VNPC6HRE8rhFIXXs1Je5s0mrDw3Uwv5BAJOL1aDgk1CkJ3XjZ5oRoY06fCMsUZRGuKUzanbx8oTdf8DnpFwr+Zn7wPERm94npoyDfTXc/RY7Zd3jrBAC0HwyvLnmw0VqNkmAGVcwMOFMiJg6WelpvvUAVO+qQrcdvnWuyMF45qQRx1sMc= 2021-03-15T01:42:20.355 INFO:teuthology.kill: smithi084.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYODzVt89gnfQFTlvU0SO8p/QtK+UZXagsA1EalyUJE860Pdm8Y9P6jEEMAdZLzoc1SIA3GsUOn1A3MjtNPphw1GKZZvJij26GCp4AKOiF32pSSglzfGN2qt8usWi8RTFYkJS7PMdZIliFzFfKcCjPKAvP32Md8JAtI3Bj51DBgcnsmgSgR1VPkadHs7b4H4rl2qL32Opoe9iZqW8ZjQqSTC0g4xlH76BbzyzDCFC+KC46fOB1xyZZdd4LKwN8Kiy+WsqM4W2q89iPHPgEBnJ9ZzeE2wMO0bTStO6lwX319ypuK57mXrcaN+0kNEgngE+RcFOm/d+wHT8lQUCyYCWc9oSajtDOsQN7qYXsviY5wrmFouuKo4iGXmGUDnMJrzSOkyE9HO04baWJxDwqGUNVAV25yK7lPkxuMWkRCHoZLXUYjZtL/oRA+0IPtQ6Rwfjo2mDgzwMKkBq74VNVb8GLvEDAE5RJTiKNpZHySaFgTLDTNBtZNCl+RHKn4R0jnFk= 2021-03-15T01:42:20.355 INFO:teuthology.kill:2021-03-15 01:42:20,350.350 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-15T01:42:20.356 INFO:teuthology.kill:2021-03-15 01:42:20,353.353 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-15T01:42:20.424 INFO:teuthology.kill:2021-03-15 01:42:20,423.423 INFO:teuthology.orchestra.console:Power off smithi061 2021-03-15T01:42:20.450 INFO:teuthology.kill:2021-03-15 01:42:20,450.450 INFO:teuthology.orchestra.console:Power off smithi084 2021-03-15T01:42:28.844 INFO:teuthology.kill:2021-03-15 01:42:28,843.843 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2021-03-15T01:42:32.945 INFO:teuthology.kill:2021-03-15 01:42:32,945.945 INFO:teuthology.orchestra.console:Power off for smithi061 completed 2021-03-15T01:42:33.189 INFO:teuthology.task.internal:roles: ubuntu@smithi061.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-03-15T01:42:33.190 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-03-15T01:42:33.190 INFO:teuthology.misc:Compressing logs... 2021-03-15T01:42:33.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-15T01:43:33.194 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-15T01:43:33.257 INFO:teuthology.kill:No teuthology processes running 2021-03-15T01:43:35.594 INFO:teuthology.kill:Nuking machines: ['smithi061', 'smithi084'] 2021-03-15T01:43:37.757 INFO:teuthology.kill:2021-03-15 01:43:37,757.757 INFO:teuthology.nuke:targets: 2021-03-15T01:43:37.762 INFO:teuthology.kill: smithi061.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVSC0xCfgsUYukMwgp3J4jX5sD7v3YZOiJ8x5MBX5UCPcUf1sXviNfnRTRx+ZhM5vPurglLGK111n2IXeWAtlYJnWTbLTW/beU44gen5lZE6aX4Ljk1eZBGfvUwDY7H2XRZse9OZghzVQtQHQhDhTwPUqb+7BmrfGA9z6CPnq+ycECdf9WAcB4nvKt+BJzd1v2CPLx3csqKH2YK1GvT0Vxr+CDG4YkitrCqe2LgHM99rQAh1Jb80FRIW1tTAETWDV68qxT5uXyvBTUCkM/qLtuFr8/Sh2DfMXKbIMkc+7P5zyFcp/9efTEkzPxvnVNGRJDulfiKE6w/z+/G5VNPC6HRE8rhFIXXs1Je5s0mrDw3Uwv5BAJOL1aDgk1CkJ3XjZ5oRoY06fCMsUZRGuKUzanbx8oTdf8DnpFwr+Zn7wPERm94npoyDfTXc/RY7Zd3jrBAC0HwyvLnmw0VqNkmAGVcwMOFMiJg6WelpvvUAVO+qQrcdvnWuyMF45qQRx1sMc= 2021-03-15T01:43:37.762 INFO:teuthology.kill: smithi084.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYODzVt89gnfQFTlvU0SO8p/QtK+UZXagsA1EalyUJE860Pdm8Y9P6jEEMAdZLzoc1SIA3GsUOn1A3MjtNPphw1GKZZvJij26GCp4AKOiF32pSSglzfGN2qt8usWi8RTFYkJS7PMdZIliFzFfKcCjPKAvP32Md8JAtI3Bj51DBgcnsmgSgR1VPkadHs7b4H4rl2qL32Opoe9iZqW8ZjQqSTC0g4xlH76BbzyzDCFC+KC46fOB1xyZZdd4LKwN8Kiy+WsqM4W2q89iPHPgEBnJ9ZzeE2wMO0bTStO6lwX319ypuK57mXrcaN+0kNEgngE+RcFOm/d+wHT8lQUCyYCWc9oSajtDOsQN7qYXsviY5wrmFouuKo4iGXmGUDnMJrzSOkyE9HO04baWJxDwqGUNVAV25yK7lPkxuMWkRCHoZLXUYjZtL/oRA+0IPtQ6Rwfjo2mDgzwMKkBq74VNVb8GLvEDAE5RJTiKNpZHySaFgTLDTNBtZNCl+RHKn4R0jnFk= 2021-03-15T01:43:37.781 INFO:teuthology.kill:2021-03-15 01:43:37,781.781 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-15T01:43:37.785 INFO:teuthology.kill:2021-03-15 01:43:37,785.785 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-15T01:43:37.850 INFO:teuthology.kill:2021-03-15 01:43:37,850.850 INFO:teuthology.orchestra.console:Power off smithi061 2021-03-15T01:43:37.869 INFO:teuthology.kill:2021-03-15 01:43:37,868.868 INFO:teuthology.orchestra.console:Power off smithi084 2021-03-15T01:43:38.020 INFO:teuthology.kill:2021-03-15 01:43:38,019.019 INFO:teuthology.orchestra.console:Power off for smithi061 completed 2021-03-15T01:43:38.031 INFO:teuthology.kill:2021-03-15 01:43:38,030.030 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2021-03-15T01:43:38.166 INFO:teuthology.kill:2021-03-15 01:43:38,165.165 INFO:teuthology.lock.ops:unlocked smithi061.front.sepia.ceph.com 2021-03-15T01:43:38.170 INFO:teuthology.kill:2021-03-15 01:43:38,170.170 INFO:teuthology.lock.ops:unlocked smithi084.front.sepia.ceph.com 2021-03-15T01:45:38.384 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-03-15T01:45:38.479 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-03-15T01:45:38.480 INFO:teuthology.nuke:Checking targets against current locks 2021-03-15T01:45:38.559 INFO:teuthology.nuke:Not nuking smithi061.front.sepia.ceph.com because description doesn't match 2021-03-15T01:45:38.560 INFO:teuthology.nuke:Not nuking smithi084.front.sepia.ceph.com because description doesn't match