2021-05-26T03:57:52.050 INFO:root:teuthology version: 1.1.0-cc3bac84 2021-05-26T03:57:52.138 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba034.front.sepia.ceph.com' reimaging 2021-05-26T03:57:52.139 INFO:teuthology.lock.ops:Updating [ubuntu@gibba034.front.sepia.ceph.com]: reset os type and version on server 2021-05-26T03:57:52.139 INFO:teuthology.lock.ops:Updating gibba034.front.sepia.ceph.com on lock server 2021-05-26T03:57:52.154 INFO:teuthology.lock.ops:Node 'ubuntu@gibba034.front.sepia.ceph.com' reimaging is complete 2021-05-26T03:57:52.155 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba038.front.sepia.ceph.com' reimaging 2021-05-26T03:57:52.155 INFO:teuthology.lock.ops:Updating [ubuntu@gibba038.front.sepia.ceph.com]: reset os type and version on server 2021-05-26T03:57:52.155 INFO:teuthology.lock.ops:Updating gibba038.front.sepia.ceph.com on lock server 2021-05-26T03:57:52.170 INFO:teuthology.lock.ops:Node 'ubuntu@gibba038.front.sepia.ceph.com' reimaging is complete 2021-05-26T03:57:52.351 INFO:teuthology.provision.fog.gibba034:Scheduling deploy of ubuntu 20.04 2021-05-26T03:57:52.358 INFO:teuthology.provision.fog.gibba038:Scheduling deploy of ubuntu 20.04 2021-05-26T03:57:52.704 INFO:teuthology.orchestra.console:Power off gibba034 2021-05-26T03:57:52.722 INFO:teuthology.orchestra.console:Power off gibba038 2021-05-26T03:58:01.092 INFO:teuthology.orchestra.console:Power off for gibba034 completed 2021-05-26T03:58:01.103 INFO:teuthology.orchestra.console:Power off for gibba038 completed 2021-05-26T03:58:01.193 INFO:teuthology.orchestra.console:Power on gibba034 2021-05-26T03:58:01.204 INFO:teuthology.orchestra.console:Power on gibba038 2021-05-26T03:58:09.575 INFO:teuthology.orchestra.console:Power on for gibba034 completed 2021-05-26T03:58:09.581 INFO:teuthology.orchestra.console:Power on for gibba038 completed 2021-05-26T03:58:09.677 INFO:teuthology.provision.fog.gibba034:Waiting for deploy to finish 2021-05-26T03:58:09.682 INFO:teuthology.provision.fog.gibba038:Waiting for deploy to finish 2021-05-26T04:02:52.156 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-26T04:02:52.647 INFO:teuthology.provision.fog.gibba034:Node is ready 2021-05-26T04:02:52.654 INFO:teuthology.orchestra.run.gibba034.stdout:gibba034.front.sepia.ceph.com 2021-05-26T04:02:52.663 INFO:teuthology.orchestra.run.gibba034.stdout:172.21.2.134 gibba034.front.sepia.ceph.com gibba034 2021-05-26T04:02:52.775 INFO:teuthology.provision.fog.gibba034:Deploy complete! 2021-05-26T04:02:59.580 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-26T04:02:59.971 INFO:teuthology.provision.fog.gibba038:Node is ready 2021-05-26T04:02:59.976 INFO:teuthology.orchestra.run.gibba038.stdout:gibba038.front.sepia.ceph.com 2021-05-26T04:02:59.981 INFO:teuthology.orchestra.run.gibba038.stdout:172.21.2.138 gibba038.front.sepia.ceph.com gibba038 2021-05-26T04:03:00.209 INFO:teuthology.provision.fog.gibba038:Deploy complete! 2021-05-26T04:03:00.335 INFO:teuthology.lock.ops:Checking gibba034.front.sepia.ceph.com 2021-05-26T04:03:00.336 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-26T04:03:00.351 INFO:teuthology.lock.ops:Checking gibba038.front.sepia.ceph.com 2021-05-26T04:03:00.351 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-26T04:03:00.366 INFO:teuthology.lock.ops:Updating [gibba034.front.sepia.ceph.com]: set os type and version on server 2021-05-26T04:03:00.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-26T04:03:00.485 INFO:teuthology.orchestra.run.gibba034.stdout:x86_64 2021-05-26T04:03:00.490 INFO:teuthology.orchestra.run.gibba034.stdout:NAME="Ubuntu" 2021-05-26T04:03:00.490 INFO:teuthology.orchestra.run.gibba034.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-05-26T04:03:00.490 INFO:teuthology.orchestra.run.gibba034.stdout:ID=ubuntu 2021-05-26T04:03:00.491 INFO:teuthology.orchestra.run.gibba034.stdout:ID_LIKE=debian 2021-05-26T04:03:00.491 INFO:teuthology.orchestra.run.gibba034.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-05-26T04:03:00.491 INFO:teuthology.orchestra.run.gibba034.stdout:VERSION_ID="20.04" 2021-05-26T04:03:00.491 INFO:teuthology.orchestra.run.gibba034.stdout:HOME_URL="https://www.ubuntu.com/" 2021-05-26T04:03:00.492 INFO:teuthology.orchestra.run.gibba034.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-05-26T04:03:00.492 INFO:teuthology.orchestra.run.gibba034.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-05-26T04:03:00.492 INFO:teuthology.orchestra.run.gibba034.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-05-26T04:03:00.492 INFO:teuthology.orchestra.run.gibba034.stdout:VERSION_CODENAME=focal 2021-05-26T04:03:00.493 INFO:teuthology.orchestra.run.gibba034.stdout:UBUNTU_CODENAME=focal 2021-05-26T04:03:00.494 INFO:teuthology.lock.ops:Updating gibba034.front.sepia.ceph.com on lock server 2021-05-26T04:03:00.508 INFO:teuthology.lock.ops:Updating [gibba038.front.sepia.ceph.com]: set os type and version on server 2021-05-26T04:03:00.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-26T04:03:00.671 INFO:teuthology.orchestra.run.gibba038.stdout:x86_64 2021-05-26T04:03:00.675 INFO:teuthology.orchestra.run.gibba038.stdout:NAME="Ubuntu" 2021-05-26T04:03:00.675 INFO:teuthology.orchestra.run.gibba038.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-05-26T04:03:00.675 INFO:teuthology.orchestra.run.gibba038.stdout:ID=ubuntu 2021-05-26T04:03:00.676 INFO:teuthology.orchestra.run.gibba038.stdout:ID_LIKE=debian 2021-05-26T04:03:00.676 INFO:teuthology.orchestra.run.gibba038.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-05-26T04:03:00.676 INFO:teuthology.orchestra.run.gibba038.stdout:VERSION_ID="20.04" 2021-05-26T04:03:00.676 INFO:teuthology.orchestra.run.gibba038.stdout:HOME_URL="https://www.ubuntu.com/" 2021-05-26T04:03:00.677 INFO:teuthology.orchestra.run.gibba038.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-05-26T04:03:00.677 INFO:teuthology.orchestra.run.gibba038.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-05-26T04:03:00.677 INFO:teuthology.orchestra.run.gibba038.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-05-26T04:03:00.677 INFO:teuthology.orchestra.run.gibba038.stdout:VERSION_CODENAME=focal 2021-05-26T04:03:00.678 INFO:teuthology.orchestra.run.gibba038.stdout:UBUNTU_CODENAME=focal 2021-05-26T04:03:00.678 INFO:teuthology.lock.ops:Updating gibba038.front.sepia.ceph.com on lock server 2021-05-26T04:03:00.755 INFO:teuthology.dispatcher.supervisor:Running job 6117739 2021-05-26T04:03:00.755 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_cc3bac844fd368a871b19f94cc5d520695b86d44/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-05-17_02:01:03-rbd-master-distro-basic-gibba/6117739 --name teuthology-2021-05-17_02:01:03-rbd-master-distro-basic-gibba --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} -- /home/teuthworker/archive/teuthology-2021-05-17_02:01:03-rbd-master-distro-basic-gibba/6117739/orig.config.yaml 2021-05-26T04:03:00.759 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-05-17_02:01:03-rbd-master-distro-basic-gibba/6117739 2021-05-26T04:03:00.760 INFO:teuthology.dispatcher.supervisor:Job PID: 29948 2021-05-26T04:03:00.760 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-26T16:03:49.900 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-26T16:03:50.154 INFO:teuthology.kill:Killing Pids: {29948} 2021-05-26T16:03:51.288 INFO:teuthology.kill:Nuking machines: ['gibba034', 'gibba038'] 2021-05-26T16:03:52.237 INFO:teuthology.kill:2021-05-26 16:03:52,237.237 INFO:teuthology.nuke:targets: 2021-05-26T16:03:52.238 INFO:teuthology.kill: gibba034.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxTqIlQOzxns7mDXB+fXlDz6OwTymKTl3EAc0RVe8o1w2nCvzqiBp24Q4gmQpSh1X6nIrceXvjB+zaSZlZzcMNcIheLMh7zvUfTij9AiIS2NgeOVJvueO8JVTKuwTMNyaRbXcPevIZWVB+sHafRUqWpfRGY4pFbiMYJIH76NYTLBrn9RZv2HT8ffp9Xjj27ShQMszLvlkGIMQYFlH61SC+Ezg2ylomNsZJdvvb2aRjmo8EIryZXAFHJlrZjNjodfrR1ZVL9Z9eHju57LwE18BbkXNKXkre9KQa4lczuaWWj6GJM6j62WfqG3gdD+CdKFwHmvv14j8KJ0Ttw8xSRGdbyvXMoC4FjUTW8WzFA2eOgzzGOKqo9vc8jDDyhZ7nvTyTHjLkcdnYup6sqVsd2XfqwqM4YGrsFYAYu7+VV3/2nkR6mbnSn3iQqkHHrEjARZ3AL2gxkSSuMe/rwVnX5S2bvt0WDdYdpD/Cv4jrIJOJI9iMdpucotZMsiYWXIIUPiM= 2021-05-26T16:03:52.238 INFO:teuthology.kill: gibba038.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrX9qSIYmYMkbAJxgbMRkST7jTtkCxPh3hcG2WkRDaS2A+iZCEHN/vAQKDyqrx81cpc6PhdfDrn3j/P7E4cgfWvPKHLjegARtBCeV2M/hsnPGPduNJlku8qQfrZoXRrOrZKMC1Cm0inaMQMWcXOGbvi3oZqJKCQ3JlL/7bYf3XuM0JutopC9hO7/6x5drwrkeE7SmMw8pcEGGIJPgppw3BIb5fa5TkTpEgxix8t9/24xT0rpxsH57HrpOSzhaxpgCSAzLHPNkSxn5K0IsVXSl6a7qD99/40/5hLVBitFf09nbrTKlMrlASD38TElkj6OdKMkjyxtfZZoF9rWbHcAoD5N4HENOsBCRz5FmcrXliJaYKkOnm/jgrXRNbEwpPYQdGDknpN74YWjnMjH6nSO9k6x9Jir9fHPnkWm8uJn9NsnXbFqiJ3lHYIyFijP6IfhrBbOKZBcNUnkc1IT2GZOCBiOtdRUbI6myI6zuOuSuhT64vg9iArrALUvltVey8w9U= 2021-05-26T16:03:52.239 INFO:teuthology.kill:2021-05-26 16:03:52,237.237 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-26T16:03:52.240 INFO:teuthology.kill:2021-05-26 16:03:52,239.239 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-26T16:03:52.299 INFO:teuthology.kill:2021-05-26 16:03:52,299.299 INFO:teuthology.orchestra.console:Power off gibba034 2021-05-26T16:03:52.313 INFO:teuthology.kill:2021-05-26 16:03:52,313.313 INFO:teuthology.orchestra.console:Power off gibba038 2021-05-26T16:04:00.776 INFO:teuthology.kill:2021-05-26 16:04:00,775.775 INFO:teuthology.orchestra.console:Power off for gibba038 completed 2021-05-26T16:04:04.796 INFO:teuthology.kill:2021-05-26 16:04:04,795.795 INFO:teuthology.orchestra.console:Power off for gibba034 completed 2021-05-26T16:04:05.037 INFO:teuthology.task.internal:roles: ubuntu@gibba034.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2021-05-26T16:04:05.038 INFO:teuthology.task.internal:roles: ubuntu@gibba038.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2021-05-26T16:04:05.038 INFO:teuthology.misc:Compressing logs... 2021-05-26T16:04:05.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-26T16:04:39.503 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_cc3bac844fd368a871b19f94cc5d520695b86d44/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_cc3bac844fd368a871b19f94cc5d520695b86d44/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_cc3bac844fd368a871b19f94cc5d520695b86d44/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_cc3bac844fd368a871b19f94cc5d520695b86d44/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_cc3bac844fd368a871b19f94cc5d520695b86d44/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_cc3bac844fd368a871b19f94cc5d520695b86d44/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_cc3bac844fd368a871b19f94cc5d520695b86d44/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-26T16:04:39.574 INFO:teuthology.kill:No teuthology processes running 2021-05-26T16:04:40.628 INFO:teuthology.kill:Nuking machines: ['gibba034', 'gibba038'] 2021-05-26T16:04:41.572 INFO:teuthology.kill:2021-05-26 16:04:41,571.571 INFO:teuthology.nuke:targets: 2021-05-26T16:04:41.573 INFO:teuthology.kill: gibba034.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxTqIlQOzxns7mDXB+fXlDz6OwTymKTl3EAc0RVe8o1w2nCvzqiBp24Q4gmQpSh1X6nIrceXvjB+zaSZlZzcMNcIheLMh7zvUfTij9AiIS2NgeOVJvueO8JVTKuwTMNyaRbXcPevIZWVB+sHafRUqWpfRGY4pFbiMYJIH76NYTLBrn9RZv2HT8ffp9Xjj27ShQMszLvlkGIMQYFlH61SC+Ezg2ylomNsZJdvvb2aRjmo8EIryZXAFHJlrZjNjodfrR1ZVL9Z9eHju57LwE18BbkXNKXkre9KQa4lczuaWWj6GJM6j62WfqG3gdD+CdKFwHmvv14j8KJ0Ttw8xSRGdbyvXMoC4FjUTW8WzFA2eOgzzGOKqo9vc8jDDyhZ7nvTyTHjLkcdnYup6sqVsd2XfqwqM4YGrsFYAYu7+VV3/2nkR6mbnSn3iQqkHHrEjARZ3AL2gxkSSuMe/rwVnX5S2bvt0WDdYdpD/Cv4jrIJOJI9iMdpucotZMsiYWXIIUPiM= 2021-05-26T16:04:41.573 INFO:teuthology.kill: gibba038.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrX9qSIYmYMkbAJxgbMRkST7jTtkCxPh3hcG2WkRDaS2A+iZCEHN/vAQKDyqrx81cpc6PhdfDrn3j/P7E4cgfWvPKHLjegARtBCeV2M/hsnPGPduNJlku8qQfrZoXRrOrZKMC1Cm0inaMQMWcXOGbvi3oZqJKCQ3JlL/7bYf3XuM0JutopC9hO7/6x5drwrkeE7SmMw8pcEGGIJPgppw3BIb5fa5TkTpEgxix8t9/24xT0rpxsH57HrpOSzhaxpgCSAzLHPNkSxn5K0IsVXSl6a7qD99/40/5hLVBitFf09nbrTKlMrlASD38TElkj6OdKMkjyxtfZZoF9rWbHcAoD5N4HENOsBCRz5FmcrXliJaYKkOnm/jgrXRNbEwpPYQdGDknpN74YWjnMjH6nSO9k6x9Jir9fHPnkWm8uJn9NsnXbFqiJ3lHYIyFijP6IfhrBbOKZBcNUnkc1IT2GZOCBiOtdRUbI6myI6zuOuSuhT64vg9iArrALUvltVey8w9U= 2021-05-26T16:04:41.592 INFO:teuthology.kill:2021-05-26 16:04:41,592.592 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-26T16:04:41.595 INFO:teuthology.kill:2021-05-26 16:04:41,595.595 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-26T16:04:41.645 INFO:teuthology.kill:2021-05-26 16:04:41,645.645 INFO:teuthology.orchestra.console:Power off gibba034 2021-05-26T16:04:41.657 INFO:teuthology.kill:2021-05-26 16:04:41,657.657 INFO:teuthology.orchestra.console:Power off gibba038 2021-05-26T16:04:41.791 INFO:teuthology.kill:2021-05-26 16:04:41,791.791 INFO:teuthology.orchestra.console:Power off for gibba034 completed 2021-05-26T16:04:41.806 INFO:teuthology.kill:2021-05-26 16:04:41,805.805 INFO:teuthology.orchestra.console:Power off for gibba038 completed 2021-05-26T16:04:41.929 INFO:teuthology.kill:2021-05-26 16:04:41,928.928 INFO:teuthology.lock.ops:unlocked gibba034.front.sepia.ceph.com 2021-05-26T16:04:41.943 INFO:teuthology.kill:2021-05-26 16:04:41,943.943 INFO:teuthology.lock.ops:unlocked gibba038.front.sepia.ceph.com 2021-05-26T16:06:42.245 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-05-26T16:06:42.345 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-05-26T16:06:42.345 INFO:teuthology.nuke:Checking targets against current locks 2021-05-26T16:06:42.421 INFO:teuthology.nuke:Not nuking gibba038.front.sepia.ceph.com because description doesn't match 2021-05-26T16:06:42.422 INFO:teuthology.nuke:Not nuking gibba034.front.sepia.ceph.com because description doesn't match