2021-03-09T10:38:45.317 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-09T10:38:45.417 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba039.front.sepia.ceph.com' reimaging 2021-03-09T10:38:45.418 INFO:teuthology.lock.ops:Updating [ubuntu@gibba039.front.sepia.ceph.com]: reset os type and version on server 2021-03-09T10:38:45.418 INFO:teuthology.lock.ops:Updating gibba039.front.sepia.ceph.com on lock server 2021-03-09T10:38:45.433 INFO:teuthology.lock.ops:Node 'ubuntu@gibba039.front.sepia.ceph.com' reimaging is complete 2021-03-09T10:38:45.433 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba042.front.sepia.ceph.com' reimaging 2021-03-09T10:38:45.434 INFO:teuthology.lock.ops:Updating [ubuntu@gibba042.front.sepia.ceph.com]: reset os type and version on server 2021-03-09T10:38:45.434 INFO:teuthology.lock.ops:Updating gibba042.front.sepia.ceph.com on lock server 2021-03-09T10:38:45.449 INFO:teuthology.lock.ops:Node 'ubuntu@gibba042.front.sepia.ceph.com' reimaging is complete 2021-03-09T10:38:45.636 INFO:teuthology.provision.fog.gibba039:Scheduling deploy of centos 8.2 2021-03-09T10:38:45.645 INFO:teuthology.provision.fog.gibba042:Scheduling deploy of centos 8.2 2021-03-09T10:38:45.911 INFO:teuthology.orchestra.console:Power off gibba042 2021-03-09T10:38:45.948 INFO:teuthology.orchestra.console:Power off gibba039 2021-03-09T10:38:54.336 INFO:teuthology.orchestra.console:Power off for gibba039 completed 2021-03-09T10:38:54.400 INFO:teuthology.orchestra.console:Power off for gibba042 completed 2021-03-09T10:38:54.439 INFO:teuthology.orchestra.console:Power on gibba039 2021-03-09T10:38:54.501 INFO:teuthology.orchestra.console:Power on gibba042 2021-03-09T10:39:02.815 INFO:teuthology.orchestra.console:Power on for gibba039 completed 2021-03-09T10:39:02.876 INFO:teuthology.orchestra.console:Power on for gibba042 completed 2021-03-09T10:39:02.917 INFO:teuthology.provision.fog.gibba039:Waiting for deploy to finish 2021-03-09T10:39:02.978 INFO:teuthology.provision.fog.gibba042:Waiting for deploy to finish 2021-03-09T10:41:03.781 ERROR:teuthology.orchestra.connection:Error authenticating with gibba042.front.sepia.ceph.com: Authentication failed. 2021-03-09T10:43:00.793 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-09T10:43:16.057 INFO:teuthology.provision.fog.gibba042:Node is ready 2021-03-09T10:43:16.071 INFO:teuthology.orchestra.run.gibba042.stdout:gibba042.front.sepia.ceph.com 2021-03-09T10:43:16.123 INFO:teuthology.orchestra.run.gibba042.stdout:172.21.2.142 gibba042.front.sepia.ceph.com gibba042 2021-03-09T10:43:16.450 INFO:teuthology.provision.fog.gibba042:Deploy complete! 2021-03-09T10:43:28.556 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-09T10:43:28.765 INFO:teuthology.provision.fog.gibba039:Node is ready 2021-03-09T10:43:28.817 INFO:teuthology.orchestra.run.gibba039.stdout:gibba039.front.sepia.ceph.com 2021-03-09T10:43:28.869 INFO:teuthology.orchestra.run.gibba039.stdout:172.21.2.139 gibba039.front.sepia.ceph.com gibba039 2021-03-09T10:43:29.153 INFO:teuthology.provision.fog.gibba039:Deploy complete! 2021-03-09T10:43:29.327 INFO:teuthology.lock.ops:Checking gibba039.front.sepia.ceph.com 2021-03-09T10:43:29.328 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-09T10:43:29.364 INFO:teuthology.lock.ops:Checking gibba042.front.sepia.ceph.com 2021-03-09T10:43:29.364 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-09T10:43:29.378 INFO:teuthology.lock.ops:Updating [gibba039.front.sepia.ceph.com]: set os type and version on server 2021-03-09T10:43:29.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-09T10:43:29.614 INFO:teuthology.orchestra.run.gibba039.stdout:x86_64 2021-03-09T10:43:29.668 INFO:teuthology.orchestra.run.gibba039.stdout:NAME="CentOS Linux" 2021-03-09T10:43:29.668 INFO:teuthology.orchestra.run.gibba039.stdout:VERSION="8 (Core)" 2021-03-09T10:43:29.668 INFO:teuthology.orchestra.run.gibba039.stdout:ID="centos" 2021-03-09T10:43:29.668 INFO:teuthology.orchestra.run.gibba039.stdout:ID_LIKE="rhel fedora" 2021-03-09T10:43:29.669 INFO:teuthology.orchestra.run.gibba039.stdout:VERSION_ID="8" 2021-03-09T10:43:29.669 INFO:teuthology.orchestra.run.gibba039.stdout:PLATFORM_ID="platform:el8" 2021-03-09T10:43:29.669 INFO:teuthology.orchestra.run.gibba039.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-03-09T10:43:29.669 INFO:teuthology.orchestra.run.gibba039.stdout:ANSI_COLOR="0;31" 2021-03-09T10:43:29.669 INFO:teuthology.orchestra.run.gibba039.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-03-09T10:43:29.670 INFO:teuthology.orchestra.run.gibba039.stdout:HOME_URL="https://www.centos.org/" 2021-03-09T10:43:29.670 INFO:teuthology.orchestra.run.gibba039.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-03-09T10:43:29.670 INFO:teuthology.orchestra.run.gibba039.stdout: 2021-03-09T10:43:29.670 INFO:teuthology.orchestra.run.gibba039.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-03-09T10:43:29.671 INFO:teuthology.orchestra.run.gibba039.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-03-09T10:43:29.671 INFO:teuthology.orchestra.run.gibba039.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-03-09T10:43:29.671 INFO:teuthology.orchestra.run.gibba039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-03-09T10:43:29.671 INFO:teuthology.orchestra.run.gibba039.stdout: 2021-03-09T10:43:29.672 INFO:teuthology.lock.ops:Updating gibba039.front.sepia.ceph.com on lock server 2021-03-09T10:43:29.689 INFO:teuthology.lock.ops:Updating [gibba042.front.sepia.ceph.com]: set os type and version on server 2021-03-09T10:43:29.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-09T10:43:29.927 INFO:teuthology.orchestra.run.gibba042.stdout:x86_64 2021-03-09T10:43:29.980 INFO:teuthology.orchestra.run.gibba042.stdout:NAME="CentOS Linux" 2021-03-09T10:43:29.980 INFO:teuthology.orchestra.run.gibba042.stdout:VERSION="8 (Core)" 2021-03-09T10:43:29.980 INFO:teuthology.orchestra.run.gibba042.stdout:ID="centos" 2021-03-09T10:43:29.981 INFO:teuthology.orchestra.run.gibba042.stdout:ID_LIKE="rhel fedora" 2021-03-09T10:43:29.981 INFO:teuthology.orchestra.run.gibba042.stdout:VERSION_ID="8" 2021-03-09T10:43:29.981 INFO:teuthology.orchestra.run.gibba042.stdout:PLATFORM_ID="platform:el8" 2021-03-09T10:43:29.981 INFO:teuthology.orchestra.run.gibba042.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-03-09T10:43:29.981 INFO:teuthology.orchestra.run.gibba042.stdout:ANSI_COLOR="0;31" 2021-03-09T10:43:29.982 INFO:teuthology.orchestra.run.gibba042.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-03-09T10:43:29.982 INFO:teuthology.orchestra.run.gibba042.stdout:HOME_URL="https://www.centos.org/" 2021-03-09T10:43:29.982 INFO:teuthology.orchestra.run.gibba042.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-03-09T10:43:29.982 INFO:teuthology.orchestra.run.gibba042.stdout: 2021-03-09T10:43:29.983 INFO:teuthology.orchestra.run.gibba042.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-03-09T10:43:29.983 INFO:teuthology.orchestra.run.gibba042.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-03-09T10:43:29.983 INFO:teuthology.orchestra.run.gibba042.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-03-09T10:43:29.983 INFO:teuthology.orchestra.run.gibba042.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-03-09T10:43:29.983 INFO:teuthology.orchestra.run.gibba042.stdout: 2021-03-09T10:43:29.984 INFO:teuthology.lock.ops:Updating gibba042.front.sepia.ceph.com on lock server 2021-03-09T10:43:30.059 INFO:teuthology.dispatcher.supervisor:Running job 5944778 2021-03-09T10:43:30.060 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-03-08_02:01:03-rbd-master-distro-basic-gibba/5944778 --name teuthology-2021-03-08_02:01:03-rbd-master-distro-basic-gibba --description rbd/persistent-writeback-cache/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{centos_8} 4-pool/cache 5-cache-mode/rwl 6-workloads/qemu_xfstests} -- /home/teuthworker/archive/teuthology-2021-03-08_02:01:03-rbd-master-distro-basic-gibba/5944778/orig.config.yaml 2021-03-09T10:43:30.064 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-03-08_02:01:03-rbd-master-distro-basic-gibba/5944778 2021-03-09T10:43:30.065 INFO:teuthology.dispatcher.supervisor:Job PID: 28587 2021-03-09T10:43:30.065 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-03-09T22:43:47.141 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-09T22:43:47.882 INFO:teuthology.kill:Killing Pids: {28587} 2021-03-09T22:43:50.075 INFO:teuthology.kill:Nuking machines: ['gibba039', 'gibba042'] 2021-03-09T22:43:51.782 INFO:teuthology.kill:2021-03-09 22:43:51,781.781 INFO:teuthology.nuke:targets: 2021-03-09T22:43:51.786 INFO:teuthology.kill: gibba039.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUtfWPPqZ9ZJDN69DNO3A6CNWeZcLccRzcutz/1Uk2mn0LID+BHOJDHuEDXRWfOZz6rE3zrXomldLWZK8W3hEFUbZuEWeR/DZBNgF5mpIYdUJw3eGlUzwr0D3KJargf4FzGEKl1Grquu8LMf8nERT6klaHJq5pm1lZAw8Xgl5S3XBOScWxj+hujWFyGffsC9vB5iGiL+R7u/jWlPm9ABzZ8i5Qos+i56W7XpIxqQfsxMP5xNWfjc0fhT8oA0Mgl2m2rXKfLLXnOZd6k1lEATesZaKfj+qhNeuUB4PHz/z6H7fcC3Xui7tCAAXfyj6UUYFpH62VA6oqVB2VlkgiaaAJeXs6J+DEE+oz55wtVzuJ/lWzYxXnSuhsnwtpDq9s2+oVjHzz7YtPuAfkJc1nuL5H1FkTG1AICHnPK6QEpRpbTLCnYt9pVO9LzzO3JRTJ7hKfEtG1nBqZ8llQc5b2Fp/HbKwPNeKWaBLEsotVmDHXEWGglr4f2aKGqxKYHyJE2Ls= 2021-03-09T22:43:51.787 INFO:teuthology.kill: gibba042.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgDI3eZKiGwfPD/ieWScZwGjT0k5LSgdVp/MsymBOxpjjolrFifc13DeAUWvSxUaTcOi3mOMZ9P3pdFPOwp20+SVtD96FqU9+dqR0mCqbHkJ9KZGbkDVscWIJeLZY6J1uAV87f1rERWKfOEeBODsP3IMR6zEX1KNXP2nz72PAc9pt/XZdMQrXMeWxEYctVF5BuSy3tUSvB8pl9aGADAVoHNRK75Waa5zgslIVdbIdZQ3ntVlbZ+1ETRGlFp3myUt/qDgHYS9Yn220rOn5VGyiopDDQZ1rUXjzN3IGAM+sbFTCzkPgNbBnkOukl720dFZ4HT1hhpv8xXUZkG4mzxkrsYyftzQFQXRgPgr/sVr9Btzhu9JyxYIQ01F9dRhN1HQvOAKfo8wVcQ1p77a3n3wtgiRmZOZnEJpgU0K/XeL38oeyPEzNZ9esAd07Us5ErbhGQH6T1Whu0wtD2dNb9pkbapd7WEPQNhQGRVJllqGeY5CtqHC4myDZa5t5WWFc0r7s= 2021-03-09T22:43:51.787 INFO:teuthology.kill:2021-03-09 22:43:51,782.782 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-09T22:43:51.787 INFO:teuthology.kill:2021-03-09 22:43:51,785.785 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-09T22:43:51.861 INFO:teuthology.kill:2021-03-09 22:43:51,860.860 INFO:teuthology.orchestra.console:Power off gibba039 2021-03-09T22:43:51.883 INFO:teuthology.kill:2021-03-09 22:43:51,883.883 INFO:teuthology.orchestra.console:Power off gibba042 2021-03-09T22:44:00.271 INFO:teuthology.kill:2021-03-09 22:44:00,270.270 INFO:teuthology.orchestra.console:Power off for gibba039 completed 2021-03-09T22:44:04.387 INFO:teuthology.kill:2021-03-09 22:44:04,386.386 INFO:teuthology.orchestra.console:Power off for gibba042 completed 2021-03-09T22:44:04.676 INFO:teuthology.task.internal:roles: ubuntu@gibba039.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1'] 2021-03-09T22:44:04.680 INFO:teuthology.task.internal:roles: ubuntu@gibba042.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.2', 'osd.3', 'client.0'] 2021-03-09T22:44:04.687 INFO:teuthology.misc:Compressing logs... 2021-03-09T22:44:04.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-09T22:44:23.373 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-09T22:44:23.441 INFO:teuthology.kill:No teuthology processes running 2021-03-09T22:44:25.124 INFO:teuthology.kill:Nuking machines: ['gibba039', 'gibba042'] 2021-03-09T22:44:26.032 INFO:teuthology.kill:2021-03-09 22:44:26,031.031 INFO:teuthology.nuke:targets: 2021-03-09T22:44:26.033 INFO:teuthology.kill: gibba039.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUtfWPPqZ9ZJDN69DNO3A6CNWeZcLccRzcutz/1Uk2mn0LID+BHOJDHuEDXRWfOZz6rE3zrXomldLWZK8W3hEFUbZuEWeR/DZBNgF5mpIYdUJw3eGlUzwr0D3KJargf4FzGEKl1Grquu8LMf8nERT6klaHJq5pm1lZAw8Xgl5S3XBOScWxj+hujWFyGffsC9vB5iGiL+R7u/jWlPm9ABzZ8i5Qos+i56W7XpIxqQfsxMP5xNWfjc0fhT8oA0Mgl2m2rXKfLLXnOZd6k1lEATesZaKfj+qhNeuUB4PHz/z6H7fcC3Xui7tCAAXfyj6UUYFpH62VA6oqVB2VlkgiaaAJeXs6J+DEE+oz55wtVzuJ/lWzYxXnSuhsnwtpDq9s2+oVjHzz7YtPuAfkJc1nuL5H1FkTG1AICHnPK6QEpRpbTLCnYt9pVO9LzzO3JRTJ7hKfEtG1nBqZ8llQc5b2Fp/HbKwPNeKWaBLEsotVmDHXEWGglr4f2aKGqxKYHyJE2Ls= 2021-03-09T22:44:26.033 INFO:teuthology.kill: gibba042.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgDI3eZKiGwfPD/ieWScZwGjT0k5LSgdVp/MsymBOxpjjolrFifc13DeAUWvSxUaTcOi3mOMZ9P3pdFPOwp20+SVtD96FqU9+dqR0mCqbHkJ9KZGbkDVscWIJeLZY6J1uAV87f1rERWKfOEeBODsP3IMR6zEX1KNXP2nz72PAc9pt/XZdMQrXMeWxEYctVF5BuSy3tUSvB8pl9aGADAVoHNRK75Waa5zgslIVdbIdZQ3ntVlbZ+1ETRGlFp3myUt/qDgHYS9Yn220rOn5VGyiopDDQZ1rUXjzN3IGAM+sbFTCzkPgNbBnkOukl720dFZ4HT1hhpv8xXUZkG4mzxkrsYyftzQFQXRgPgr/sVr9Btzhu9JyxYIQ01F9dRhN1HQvOAKfo8wVcQ1p77a3n3wtgiRmZOZnEJpgU0K/XeL38oeyPEzNZ9esAd07Us5ErbhGQH6T1Whu0wtD2dNb9pkbapd7WEPQNhQGRVJllqGeY5CtqHC4myDZa5t5WWFc0r7s= 2021-03-09T22:44:26.053 INFO:teuthology.kill:2021-03-09 22:44:26,052.052 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-09T22:44:26.056 INFO:teuthology.kill:2021-03-09 22:44:26,055.055 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-09T22:44:26.117 INFO:teuthology.kill:2021-03-09 22:44:26,117.117 INFO:teuthology.orchestra.console:Power off gibba039 2021-03-09T22:44:26.131 INFO:teuthology.kill:2021-03-09 22:44:26,131.131 INFO:teuthology.orchestra.console:Power off gibba042 2021-03-09T22:44:26.266 INFO:teuthology.kill:2021-03-09 22:44:26,265.265 INFO:teuthology.orchestra.console:Power off for gibba039 completed 2021-03-09T22:44:26.277 INFO:teuthology.kill:2021-03-09 22:44:26,277.277 INFO:teuthology.orchestra.console:Power off for gibba042 completed 2021-03-09T22:44:26.406 INFO:teuthology.kill:2021-03-09 22:44:26,405.405 INFO:teuthology.lock.ops:unlocked gibba039.front.sepia.ceph.com 2021-03-09T22:44:26.417 INFO:teuthology.kill:2021-03-09 22:44:26,416.416 INFO:teuthology.lock.ops:unlocked gibba042.front.sepia.ceph.com 2021-03-09T22:46:26.601 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-03-09T22:46:26.687 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-03-09T22:46:26.690 INFO:teuthology.nuke:Checking targets against current locks 2021-03-09T22:46:27.020 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/scripts/dispatcher.py", line 33, in main teuthology.dispatcher.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/dispatcher/supervisor.py", line 59, in main verbose File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/dispatcher/supervisor.py", line 152, in run_job unlock_targets(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/dispatcher/supervisor.py", line 196, in unlock_targets nuke(fake_ctx, True) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable