2021-03-20T23:48:50.360 INFO:root:teuthology version: 1.1.0-b6eecfe8 2021-03-20T23:48:50.528 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba002.front.sepia.ceph.com' reimaging 2021-03-20T23:48:50.529 INFO:teuthology.lock.ops:Updating [ubuntu@gibba002.front.sepia.ceph.com]: reset os type and version on server 2021-03-20T23:48:50.529 INFO:teuthology.lock.ops:Updating gibba002.front.sepia.ceph.com on lock server 2021-03-20T23:48:50.546 INFO:teuthology.lock.ops:Node 'ubuntu@gibba002.front.sepia.ceph.com' reimaging is complete 2021-03-20T23:48:50.547 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba023.front.sepia.ceph.com' reimaging 2021-03-20T23:48:50.547 INFO:teuthology.lock.ops:Updating [ubuntu@gibba023.front.sepia.ceph.com]: reset os type and version on server 2021-03-20T23:48:50.547 INFO:teuthology.lock.ops:Updating gibba023.front.sepia.ceph.com on lock server 2021-03-20T23:48:50.563 INFO:teuthology.lock.ops:Node 'ubuntu@gibba023.front.sepia.ceph.com' reimaging is complete 2021-03-20T23:48:50.563 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba025.front.sepia.ceph.com' reimaging 2021-03-20T23:48:50.563 INFO:teuthology.lock.ops:Updating [ubuntu@gibba025.front.sepia.ceph.com]: reset os type and version on server 2021-03-20T23:48:50.563 INFO:teuthology.lock.ops:Updating gibba025.front.sepia.ceph.com on lock server 2021-03-20T23:48:50.582 INFO:teuthology.lock.ops:Node 'ubuntu@gibba025.front.sepia.ceph.com' reimaging is complete 2021-03-20T23:48:50.583 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba028.front.sepia.ceph.com' reimaging 2021-03-20T23:48:50.583 INFO:teuthology.lock.ops:Updating [ubuntu@gibba028.front.sepia.ceph.com]: reset os type and version on server 2021-03-20T23:48:50.583 INFO:teuthology.lock.ops:Updating gibba028.front.sepia.ceph.com on lock server 2021-03-20T23:48:50.596 INFO:teuthology.lock.ops:Node 'ubuntu@gibba028.front.sepia.ceph.com' reimaging is complete 2021-03-20T23:48:50.596 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba040.front.sepia.ceph.com' reimaging 2021-03-20T23:48:50.596 INFO:teuthology.lock.ops:Updating [ubuntu@gibba040.front.sepia.ceph.com]: reset os type and version on server 2021-03-20T23:48:50.597 INFO:teuthology.lock.ops:Updating gibba040.front.sepia.ceph.com on lock server 2021-03-20T23:48:50.611 INFO:teuthology.lock.ops:Node 'ubuntu@gibba040.front.sepia.ceph.com' reimaging is complete 2021-03-20T23:48:50.771 INFO:teuthology.provision.fog.gibba002:Scheduling deploy of ubuntu 16.04 2021-03-20T23:48:50.810 INFO:teuthology.provision.fog.gibba025:Scheduling deploy of ubuntu 16.04 2021-03-20T23:48:50.812 INFO:teuthology.provision.fog.gibba023:Scheduling deploy of ubuntu 16.04 2021-03-20T23:48:50.818 INFO:teuthology.provision.fog.gibba028:Scheduling deploy of ubuntu 16.04 2021-03-20T23:48:50.862 INFO:teuthology.provision.fog.gibba040:Scheduling deploy of ubuntu 16.04 2021-03-20T23:48:51.128 INFO:teuthology.orchestra.console:Power off gibba002 2021-03-20T23:48:51.145 INFO:teuthology.orchestra.console:Power off gibba028 2021-03-20T23:48:51.158 INFO:teuthology.orchestra.console:Power off gibba023 2021-03-20T23:48:51.176 INFO:teuthology.orchestra.console:Power off gibba025 2021-03-20T23:48:51.255 INFO:teuthology.orchestra.console:Power off gibba040 2021-03-20T23:48:59.520 INFO:teuthology.orchestra.console:Power off for gibba002 completed 2021-03-20T23:48:59.525 INFO:teuthology.orchestra.console:Power off for gibba023 completed 2021-03-20T23:48:59.545 INFO:teuthology.orchestra.console:Power off for gibba025 completed 2021-03-20T23:48:59.608 INFO:teuthology.orchestra.console:Power off for gibba028 completed 2021-03-20T23:48:59.618 INFO:teuthology.orchestra.console:Power off for gibba040 completed 2021-03-20T23:48:59.621 INFO:teuthology.orchestra.console:Power on gibba002 2021-03-20T23:48:59.627 INFO:teuthology.orchestra.console:Power on gibba023 2021-03-20T23:48:59.648 INFO:teuthology.orchestra.console:Power on gibba025 2021-03-20T23:48:59.709 INFO:teuthology.orchestra.console:Power on gibba028 2021-03-20T23:48:59.719 INFO:teuthology.orchestra.console:Power on gibba040 2021-03-20T23:49:07.992 INFO:teuthology.orchestra.console:Power on for gibba002 completed 2021-03-20T23:49:07.997 INFO:teuthology.orchestra.console:Power on for gibba023 completed 2021-03-20T23:49:08.011 INFO:teuthology.orchestra.console:Power on for gibba025 completed 2021-03-20T23:49:08.071 INFO:teuthology.orchestra.console:Power on for gibba028 completed 2021-03-20T23:49:08.081 INFO:teuthology.orchestra.console:Power on for gibba040 completed 2021-03-20T23:49:08.093 INFO:teuthology.provision.fog.gibba002:Waiting for deploy to finish 2021-03-20T23:49:08.097 INFO:teuthology.provision.fog.gibba023:Waiting for deploy to finish 2021-03-20T23:49:08.111 INFO:teuthology.provision.fog.gibba025:Waiting for deploy to finish 2021-03-20T23:49:08.173 INFO:teuthology.provision.fog.gibba028:Waiting for deploy to finish 2021-03-20T23:49:08.182 INFO:teuthology.provision.fog.gibba040:Waiting for deploy to finish 2021-03-20T23:53:05.482 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-03-20T23:53:05.484 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-03-20T23:53:05.489 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-03-20T23:53:05.609 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-03-20T23:53:11.493 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-03-20T23:53:11.615 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-03-20T23:53:17.620 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-03-20T23:53:23.611 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-20T23:53:23.626 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-20T23:53:23.793 INFO:teuthology.provision.fog.gibba002:Node is ready 2021-03-20T23:53:23.816 INFO:teuthology.provision.fog.gibba040:Node is ready 2021-03-20T23:53:23.833 INFO:teuthology.orchestra.run.gibba002.stdout:gibba002.front.sepia.ceph.com 2021-03-20T23:53:23.855 INFO:teuthology.orchestra.run.gibba040.stdout:gibba040.front.sepia.ceph.com 2021-03-20T23:53:23.873 INFO:teuthology.orchestra.run.gibba002.stdout:172.21.2.102 gibba002.front.sepia.ceph.com gibba002 2021-03-20T23:53:23.896 INFO:teuthology.orchestra.run.gibba040.stdout:172.21.2.140 gibba040.front.sepia.ceph.com gibba040 2021-03-20T23:53:24.110 INFO:teuthology.provision.fog.gibba002:Deploy complete! 2021-03-20T23:53:24.145 INFO:teuthology.provision.fog.gibba040:Deploy complete! 2021-03-20T23:53:26.759 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-20T23:53:26.996 INFO:teuthology.provision.fog.gibba028:Node is ready 2021-03-20T23:53:27.035 INFO:teuthology.orchestra.run.gibba028.stdout:gibba028.front.sepia.ceph.com 2021-03-20T23:53:27.077 INFO:teuthology.orchestra.run.gibba028.stdout:172.21.2.128 gibba028.front.sepia.ceph.com gibba028 2021-03-20T23:53:27.317 INFO:teuthology.provision.fog.gibba028:Deploy complete! 2021-03-20T23:53:29.625 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-20T23:53:29.825 INFO:teuthology.provision.fog.gibba023:Node is ready 2021-03-20T23:53:29.863 INFO:teuthology.orchestra.run.gibba023.stdout:gibba023.front.sepia.ceph.com 2021-03-20T23:53:29.904 INFO:teuthology.orchestra.run.gibba023.stdout:172.21.2.123 gibba023.front.sepia.ceph.com gibba023 2021-03-20T23:53:30.118 INFO:teuthology.provision.fog.gibba023:Deploy complete! 2021-03-20T23:53:39.949 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-20T23:53:40.141 INFO:teuthology.provision.fog.gibba025:Node is ready 2021-03-20T23:53:40.183 INFO:teuthology.orchestra.run.gibba025.stdout:gibba025.front.sepia.ceph.com 2021-03-20T23:53:40.222 INFO:teuthology.orchestra.run.gibba025.stdout:172.21.2.125 gibba025.front.sepia.ceph.com gibba025 2021-03-20T23:53:40.464 INFO:teuthology.provision.fog.gibba025:Deploy complete! 2021-03-20T23:53:40.633 INFO:teuthology.lock.ops:Checking gibba002.front.sepia.ceph.com 2021-03-20T23:53:40.634 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-20T23:53:40.652 INFO:teuthology.lock.ops:Checking gibba023.front.sepia.ceph.com 2021-03-20T23:53:40.652 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-20T23:53:40.664 INFO:teuthology.lock.ops:Checking gibba025.front.sepia.ceph.com 2021-03-20T23:53:40.665 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-20T23:53:40.678 INFO:teuthology.lock.ops:Checking gibba028.front.sepia.ceph.com 2021-03-20T23:53:40.678 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-20T23:53:40.690 INFO:teuthology.lock.ops:Checking gibba040.front.sepia.ceph.com 2021-03-20T23:53:40.690 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-20T23:53:40.704 INFO:teuthology.lock.ops:Updating [gibba002.front.sepia.ceph.com]: set os type and version on server 2021-03-20T23:53:40.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:53:40.997 INFO:teuthology.orchestra.run.gibba002.stdout:x86_64 2021-03-20T23:53:41.037 INFO:teuthology.orchestra.run.gibba002.stdout:NAME="Ubuntu" 2021-03-20T23:53:41.037 INFO:teuthology.orchestra.run.gibba002.stdout:VERSION="16.04.7 LTS (Xenial Xerus)" 2021-03-20T23:53:41.037 INFO:teuthology.orchestra.run.gibba002.stdout:ID=ubuntu 2021-03-20T23:53:41.038 INFO:teuthology.orchestra.run.gibba002.stdout:ID_LIKE=debian 2021-03-20T23:53:41.038 INFO:teuthology.orchestra.run.gibba002.stdout:PRETTY_NAME="Ubuntu 16.04.7 LTS" 2021-03-20T23:53:41.038 INFO:teuthology.orchestra.run.gibba002.stdout:VERSION_ID="16.04" 2021-03-20T23:53:41.038 INFO:teuthology.orchestra.run.gibba002.stdout:HOME_URL="http://www.ubuntu.com/" 2021-03-20T23:53:41.038 INFO:teuthology.orchestra.run.gibba002.stdout:SUPPORT_URL="http://help.ubuntu.com/" 2021-03-20T23:53:41.038 INFO:teuthology.orchestra.run.gibba002.stdout:BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" 2021-03-20T23:53:41.038 INFO:teuthology.orchestra.run.gibba002.stdout:VERSION_CODENAME=xenial 2021-03-20T23:53:41.038 INFO:teuthology.orchestra.run.gibba002.stdout:UBUNTU_CODENAME=xenial 2021-03-20T23:53:41.039 INFO:teuthology.lock.ops:Updating gibba002.front.sepia.ceph.com on lock server 2021-03-20T23:53:41.055 INFO:teuthology.lock.ops:Updating [gibba023.front.sepia.ceph.com]: set os type and version on server 2021-03-20T23:53:41.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:53:41.328 INFO:teuthology.orchestra.run.gibba023.stdout:x86_64 2021-03-20T23:53:41.368 INFO:teuthology.orchestra.run.gibba023.stdout:NAME="Ubuntu" 2021-03-20T23:53:41.368 INFO:teuthology.orchestra.run.gibba023.stdout:VERSION="16.04.7 LTS (Xenial Xerus)" 2021-03-20T23:53:41.368 INFO:teuthology.orchestra.run.gibba023.stdout:ID=ubuntu 2021-03-20T23:53:41.368 INFO:teuthology.orchestra.run.gibba023.stdout:ID_LIKE=debian 2021-03-20T23:53:41.368 INFO:teuthology.orchestra.run.gibba023.stdout:PRETTY_NAME="Ubuntu 16.04.7 LTS" 2021-03-20T23:53:41.369 INFO:teuthology.orchestra.run.gibba023.stdout:VERSION_ID="16.04" 2021-03-20T23:53:41.369 INFO:teuthology.orchestra.run.gibba023.stdout:HOME_URL="http://www.ubuntu.com/" 2021-03-20T23:53:41.369 INFO:teuthology.orchestra.run.gibba023.stdout:SUPPORT_URL="http://help.ubuntu.com/" 2021-03-20T23:53:41.369 INFO:teuthology.orchestra.run.gibba023.stdout:BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" 2021-03-20T23:53:41.369 INFO:teuthology.orchestra.run.gibba023.stdout:VERSION_CODENAME=xenial 2021-03-20T23:53:41.369 INFO:teuthology.orchestra.run.gibba023.stdout:UBUNTU_CODENAME=xenial 2021-03-20T23:53:41.369 INFO:teuthology.lock.ops:Updating gibba023.front.sepia.ceph.com on lock server 2021-03-20T23:53:41.387 INFO:teuthology.lock.ops:Updating [gibba025.front.sepia.ceph.com]: set os type and version on server 2021-03-20T23:53:41.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:53:41.671 INFO:teuthology.orchestra.run.gibba025.stdout:x86_64 2021-03-20T23:53:41.711 INFO:teuthology.orchestra.run.gibba025.stdout:NAME="Ubuntu" 2021-03-20T23:53:41.711 INFO:teuthology.orchestra.run.gibba025.stdout:VERSION="16.04.7 LTS (Xenial Xerus)" 2021-03-20T23:53:41.711 INFO:teuthology.orchestra.run.gibba025.stdout:ID=ubuntu 2021-03-20T23:53:41.711 INFO:teuthology.orchestra.run.gibba025.stdout:ID_LIKE=debian 2021-03-20T23:53:41.711 INFO:teuthology.orchestra.run.gibba025.stdout:PRETTY_NAME="Ubuntu 16.04.7 LTS" 2021-03-20T23:53:41.711 INFO:teuthology.orchestra.run.gibba025.stdout:VERSION_ID="16.04" 2021-03-20T23:53:41.711 INFO:teuthology.orchestra.run.gibba025.stdout:HOME_URL="http://www.ubuntu.com/" 2021-03-20T23:53:41.711 INFO:teuthology.orchestra.run.gibba025.stdout:SUPPORT_URL="http://help.ubuntu.com/" 2021-03-20T23:53:41.711 INFO:teuthology.orchestra.run.gibba025.stdout:BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" 2021-03-20T23:53:41.712 INFO:teuthology.orchestra.run.gibba025.stdout:VERSION_CODENAME=xenial 2021-03-20T23:53:41.712 INFO:teuthology.orchestra.run.gibba025.stdout:UBUNTU_CODENAME=xenial 2021-03-20T23:53:41.712 INFO:teuthology.lock.ops:Updating gibba025.front.sepia.ceph.com on lock server 2021-03-20T23:53:41.730 INFO:teuthology.lock.ops:Updating [gibba028.front.sepia.ceph.com]: set os type and version on server 2021-03-20T23:53:41.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:53:42.004 INFO:teuthology.orchestra.run.gibba028.stdout:x86_64 2021-03-20T23:53:42.044 INFO:teuthology.orchestra.run.gibba028.stdout:NAME="Ubuntu" 2021-03-20T23:53:42.045 INFO:teuthology.orchestra.run.gibba028.stdout:VERSION="16.04.7 LTS (Xenial Xerus)" 2021-03-20T23:53:42.045 INFO:teuthology.orchestra.run.gibba028.stdout:ID=ubuntu 2021-03-20T23:53:42.045 INFO:teuthology.orchestra.run.gibba028.stdout:ID_LIKE=debian 2021-03-20T23:53:42.045 INFO:teuthology.orchestra.run.gibba028.stdout:PRETTY_NAME="Ubuntu 16.04.7 LTS" 2021-03-20T23:53:42.045 INFO:teuthology.orchestra.run.gibba028.stdout:VERSION_ID="16.04" 2021-03-20T23:53:42.045 INFO:teuthology.orchestra.run.gibba028.stdout:HOME_URL="http://www.ubuntu.com/" 2021-03-20T23:53:42.045 INFO:teuthology.orchestra.run.gibba028.stdout:SUPPORT_URL="http://help.ubuntu.com/" 2021-03-20T23:53:42.045 INFO:teuthology.orchestra.run.gibba028.stdout:BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" 2021-03-20T23:53:42.045 INFO:teuthology.orchestra.run.gibba028.stdout:VERSION_CODENAME=xenial 2021-03-20T23:53:42.045 INFO:teuthology.orchestra.run.gibba028.stdout:UBUNTU_CODENAME=xenial 2021-03-20T23:53:42.046 INFO:teuthology.lock.ops:Updating gibba028.front.sepia.ceph.com on lock server 2021-03-20T23:53:42.064 INFO:teuthology.lock.ops:Updating [gibba040.front.sepia.ceph.com]: set os type and version on server 2021-03-20T23:53:42.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-20T23:53:42.348 INFO:teuthology.orchestra.run.gibba040.stdout:x86_64 2021-03-20T23:53:42.388 INFO:teuthology.orchestra.run.gibba040.stdout:NAME="Ubuntu" 2021-03-20T23:53:42.388 INFO:teuthology.orchestra.run.gibba040.stdout:VERSION="16.04.7 LTS (Xenial Xerus)" 2021-03-20T23:53:42.389 INFO:teuthology.orchestra.run.gibba040.stdout:ID=ubuntu 2021-03-20T23:53:42.389 INFO:teuthology.orchestra.run.gibba040.stdout:ID_LIKE=debian 2021-03-20T23:53:42.389 INFO:teuthology.orchestra.run.gibba040.stdout:PRETTY_NAME="Ubuntu 16.04.7 LTS" 2021-03-20T23:53:42.389 INFO:teuthology.orchestra.run.gibba040.stdout:VERSION_ID="16.04" 2021-03-20T23:53:42.389 INFO:teuthology.orchestra.run.gibba040.stdout:HOME_URL="http://www.ubuntu.com/" 2021-03-20T23:53:42.389 INFO:teuthology.orchestra.run.gibba040.stdout:SUPPORT_URL="http://help.ubuntu.com/" 2021-03-20T23:53:42.389 INFO:teuthology.orchestra.run.gibba040.stdout:BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" 2021-03-20T23:53:42.389 INFO:teuthology.orchestra.run.gibba040.stdout:VERSION_CODENAME=xenial 2021-03-20T23:53:42.389 INFO:teuthology.orchestra.run.gibba040.stdout:UBUNTU_CODENAME=xenial 2021-03-20T23:53:42.389 INFO:teuthology.lock.ops:Updating gibba040.front.sepia.ceph.com on lock server 2021-03-20T23:53:42.479 INFO:teuthology.dispatcher.supervisor:Running job 5955004 2021-03-20T23:53:42.479 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-03-11_02:30:03-upgrade:mimic-x-nautilus-distro-basic-gibba/5955004 --name teuthology-2021-03-11_02:30:03-upgrade:mimic-x-nautilus-distro-basic-gibba --description upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-nautilus 6.1-msgr2 7-final-workload/{rbd-python rgw-swift-ragweed_check snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_16.04 thrashosds-health} -- /home/teuthworker/archive/teuthology-2021-03-11_02:30:03-upgrade:mimic-x-nautilus-distro-basic-gibba/5955004/orig.config.yaml 2021-03-20T23:53:42.484 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-03-11_02:30:03-upgrade:mimic-x-nautilus-distro-basic-gibba/5955004 2021-03-20T23:53:42.484 INFO:teuthology.dispatcher.supervisor:Job PID: 10853 2021-03-20T23:53:42.484 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-03-21T11:54:30.939 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-21T11:54:31.242 INFO:teuthology.kill:Killing Pids: {10853} 2021-03-21T11:54:33.284 INFO:teuthology.kill:Nuking machines: ['gibba002', 'gibba023', 'gibba025', 'gibba028', 'gibba040'] 2021-03-21T11:54:34.239 INFO:teuthology.kill:2021-03-21 11:54:34,239.239 INFO:teuthology.nuke:targets: 2021-03-21T11:54:34.240 INFO:teuthology.kill: gibba002.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD6ktmTeEWg51Z6hMg/x2UsPNaSmficJiRxDhx9vIqx3AJODSf0DkRPtvsTyWBLYmz1DHufcQom/5BhOKm/ez0Y/jjnydP3vIH6HrPixtWWquwRPTHEIWRvyI//iLVpBb6TYqA7Fwdyp/TLKL87nk3T6MEgt4ghrf5lYCnqY1A9p28ZCJZP8R13ox1iAGqIho6be8HDbYCLFA0hBLcVLn8zghI18lCO5F55fLnyDPLuVt+zyHWTP7OmUhG1cv+L8NI9CfE7xPeRVu22CzCw+7K+5Pk53+p6oMb0oCanbfSOxu9V5EjYFvC9UEk6VnJnJRg2DWlYaRBNV9Az2+wjC2m9 2021-03-21T11:54:34.240 INFO:teuthology.kill: gibba023.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDS+N5wW4EovskqFpU6Hx0R0jcA7GqPDumBW0wTeHTJZESMzOTgu1LVeQCmsMlCqtwfw9cml+KKm4BD8qZSG6BfmuNq/ZAbFL70IYqHlKUr7PCmkGrm+RYY46Pq/Qby2OW86306VSb5zAyIsn+oJmTl5yw8Yf7Sso/M3/6sNqRcXFDf6KqKCL7ECwGVYRYfVVKszWNZ12AGUQ4iEG9HBApLZkkG/F+r11LiEbRQu12b1L7M454gVaToYoxCfRwS1xVKjni7b8/ew5fXnr9Onx1Z8GJP/41BUNRPIUGwo/LRf4THx9B6oEWMDxKZQhYjgzTYxXIvnTsNrRBX6iYDhVbt 2021-03-21T11:54:34.240 INFO:teuthology.kill: gibba025.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDonugTjr+AIMKL+D76g4dif8WHjJeKBWq6CftAQMJ5sJ827IS/WlwApznhXh7pcO7RD7kSMtOPiGlI4zLBXxmsYb0Z3+uqNscT+0EZEjP3nZo19vwLcjx3/M+9rsAbvZN0F1178uQd2Jp7CgIzLf6/Qts4+kgVb2CUHhA2QuDL/pQxhqUrqMQ8FzhrEnhWF0xcRBMKRbynI80ZYNZogixAzBl7DPNq1Ro6nJYBEKMsVM9NYaVVCGxTHIJhbLQT09S3Ixq2LnPVVTJe9dXJxbgJRggIRx5nFTA/uDPVqpitKQkz9BzEswBbBKjmSOzx2R0e0HXHChqUIzEcTYuvFxST 2021-03-21T11:54:34.240 INFO:teuthology.kill: gibba028.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCmy5DvlUGYjXy5/9EhURymDzMriuKwiMa+oOLvnumEoFeCJwqp/ra3qoaGIhMG8ugdHVWkm8nZ9tsRWP66s6FWKeZw+zRotE/VUwbUILEknZ40q3fnce8AbUMv0nadHGnQkgvqeCEvyyU8IQWvJmkdiKoJnxhoVXGznBrxdxg4mpmRU83M9w0w0wFhDT0+7AJ4Fdngsr8yEia2IPtG/BAa49zhacQscQma2Lg+Mp9/ARRLFmQG+3n1nlQr1wY9Eks+kMLcC9BmzMbLWKJq4A1dkXkuEAUJwmPb6YCrggIt8UQikS1pIEuCgMHUDsG+piRxZnl4XFxZIyM8bXiNDdF1 2021-03-21T11:54:34.240 INFO:teuthology.kill: gibba040.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4qxcIStblGuZj4iKbQMBC7MDvdWmv4G1l6Hsc2ERaKEYP6lVKF0e8fa3FybxiVw1AFt1OhhuM2zc7idriqhmQM1ReEK7+Pasn6MtTNv0rfCer+a5n4mAIRF9rcgbfwNcDIOXfc0p3O6wWTIsi3Y8XSP958eH1q8w2qDKe6dZd2NEYOy681pE618sYyLr1y+H+sYdUn+3vzgGR2O2Cbh49KwgIACoTNWPl6ugrvJxRTYux05vqQjgsWP1ClFrlqqWjynv7sc7nQknj9zEdRRr5bz0b/7RBaASguPB2R1mK5hkISlqHeGMk6IFoC2ksczIIzPXHXHxiDG7fqEyi1OvH 2021-03-21T11:54:34.241 INFO:teuthology.kill:2021-03-21 11:54:34,240.240 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-21T11:54:34.243 INFO:teuthology.kill:2021-03-21 11:54:34,243.243 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-21T11:54:34.244 INFO:teuthology.kill:2021-03-21 11:54:34,244.244 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-21T11:54:34.245 INFO:teuthology.kill:2021-03-21 11:54:34,245.245 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-21T11:54:34.247 INFO:teuthology.kill:2021-03-21 11:54:34,246.246 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-21T11:54:34.337 INFO:teuthology.kill:2021-03-21 11:54:34,336.336 INFO:teuthology.orchestra.console:Power off gibba028 2021-03-21T11:54:34.357 INFO:teuthology.kill:2021-03-21 11:54:34,357.357 INFO:teuthology.orchestra.console:Power off gibba002 2021-03-21T11:54:34.372 INFO:teuthology.kill:2021-03-21 11:54:34,372.372 INFO:teuthology.orchestra.console:Power off gibba023 2021-03-21T11:54:34.386 INFO:teuthology.kill:2021-03-21 11:54:34,386.386 INFO:teuthology.orchestra.console:Power off gibba040 2021-03-21T11:54:34.406 INFO:teuthology.kill:2021-03-21 11:54:34,406.406 INFO:teuthology.orchestra.console:Power off gibba025 2021-03-21T11:54:42.762 INFO:teuthology.kill:2021-03-21 11:54:42,762.762 INFO:teuthology.orchestra.console:Power off for gibba028 completed 2021-03-21T11:54:42.774 INFO:teuthology.kill:2021-03-21 11:54:42,773.773 INFO:teuthology.orchestra.console:Power off for gibba040 completed 2021-03-21T11:54:42.789 INFO:teuthology.kill:2021-03-21 11:54:42,789.789 INFO:teuthology.orchestra.console:Power off for gibba025 completed 2021-03-21T11:54:42.845 INFO:teuthology.kill:2021-03-21 11:54:42,844.844 INFO:teuthology.orchestra.console:Power off for gibba023 completed 2021-03-21T11:54:46.875 INFO:teuthology.kill:2021-03-21 11:54:46,875.875 INFO:teuthology.orchestra.console:Power off for gibba002 completed 2021-03-21T11:54:47.156 INFO:teuthology.task.internal:roles: ubuntu@gibba002.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2021-03-21T11:54:47.156 INFO:teuthology.task.internal:roles: ubuntu@gibba023.front.sepia.ceph.com - ['mon.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2021-03-21T11:54:47.156 INFO:teuthology.task.internal:roles: ubuntu@gibba025.front.sepia.ceph.com - ['mon.c'] 2021-03-21T11:54:47.156 INFO:teuthology.task.internal:roles: ubuntu@gibba028.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11'] 2021-03-21T11:54:47.156 INFO:teuthology.task.internal:roles: ubuntu@gibba040.front.sepia.ceph.com - ['client.0'] 2021-03-21T11:54:47.156 INFO:teuthology.misc:Compressing logs... 2021-03-21T11:54:47.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-21T11:55:47.214 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-21T11:55:47.325 INFO:teuthology.kill:No teuthology processes running 2021-03-21T11:55:48.407 INFO:teuthology.kill:Nuking machines: ['gibba002', 'gibba023', 'gibba025', 'gibba028', 'gibba040'] 2021-03-21T11:55:49.342 INFO:teuthology.kill:2021-03-21 11:55:49,342.342 INFO:teuthology.nuke:targets: 2021-03-21T11:55:49.342 INFO:teuthology.kill: gibba002.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD6ktmTeEWg51Z6hMg/x2UsPNaSmficJiRxDhx9vIqx3AJODSf0DkRPtvsTyWBLYmz1DHufcQom/5BhOKm/ez0Y/jjnydP3vIH6HrPixtWWquwRPTHEIWRvyI//iLVpBb6TYqA7Fwdyp/TLKL87nk3T6MEgt4ghrf5lYCnqY1A9p28ZCJZP8R13ox1iAGqIho6be8HDbYCLFA0hBLcVLn8zghI18lCO5F55fLnyDPLuVt+zyHWTP7OmUhG1cv+L8NI9CfE7xPeRVu22CzCw+7K+5Pk53+p6oMb0oCanbfSOxu9V5EjYFvC9UEk6VnJnJRg2DWlYaRBNV9Az2+wjC2m9 2021-03-21T11:55:49.343 INFO:teuthology.kill: gibba023.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDS+N5wW4EovskqFpU6Hx0R0jcA7GqPDumBW0wTeHTJZESMzOTgu1LVeQCmsMlCqtwfw9cml+KKm4BD8qZSG6BfmuNq/ZAbFL70IYqHlKUr7PCmkGrm+RYY46Pq/Qby2OW86306VSb5zAyIsn+oJmTl5yw8Yf7Sso/M3/6sNqRcXFDf6KqKCL7ECwGVYRYfVVKszWNZ12AGUQ4iEG9HBApLZkkG/F+r11LiEbRQu12b1L7M454gVaToYoxCfRwS1xVKjni7b8/ew5fXnr9Onx1Z8GJP/41BUNRPIUGwo/LRf4THx9B6oEWMDxKZQhYjgzTYxXIvnTsNrRBX6iYDhVbt 2021-03-21T11:55:49.343 INFO:teuthology.kill: gibba025.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDonugTjr+AIMKL+D76g4dif8WHjJeKBWq6CftAQMJ5sJ827IS/WlwApznhXh7pcO7RD7kSMtOPiGlI4zLBXxmsYb0Z3+uqNscT+0EZEjP3nZo19vwLcjx3/M+9rsAbvZN0F1178uQd2Jp7CgIzLf6/Qts4+kgVb2CUHhA2QuDL/pQxhqUrqMQ8FzhrEnhWF0xcRBMKRbynI80ZYNZogixAzBl7DPNq1Ro6nJYBEKMsVM9NYaVVCGxTHIJhbLQT09S3Ixq2LnPVVTJe9dXJxbgJRggIRx5nFTA/uDPVqpitKQkz9BzEswBbBKjmSOzx2R0e0HXHChqUIzEcTYuvFxST 2021-03-21T11:55:49.343 INFO:teuthology.kill: gibba028.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCmy5DvlUGYjXy5/9EhURymDzMriuKwiMa+oOLvnumEoFeCJwqp/ra3qoaGIhMG8ugdHVWkm8nZ9tsRWP66s6FWKeZw+zRotE/VUwbUILEknZ40q3fnce8AbUMv0nadHGnQkgvqeCEvyyU8IQWvJmkdiKoJnxhoVXGznBrxdxg4mpmRU83M9w0w0wFhDT0+7AJ4Fdngsr8yEia2IPtG/BAa49zhacQscQma2Lg+Mp9/ARRLFmQG+3n1nlQr1wY9Eks+kMLcC9BmzMbLWKJq4A1dkXkuEAUJwmPb6YCrggIt8UQikS1pIEuCgMHUDsG+piRxZnl4XFxZIyM8bXiNDdF1 2021-03-21T11:55:49.343 INFO:teuthology.kill: gibba040.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4qxcIStblGuZj4iKbQMBC7MDvdWmv4G1l6Hsc2ERaKEYP6lVKF0e8fa3FybxiVw1AFt1OhhuM2zc7idriqhmQM1ReEK7+Pasn6MtTNv0rfCer+a5n4mAIRF9rcgbfwNcDIOXfc0p3O6wWTIsi3Y8XSP958eH1q8w2qDKe6dZd2NEYOy681pE618sYyLr1y+H+sYdUn+3vzgGR2O2Cbh49KwgIACoTNWPl6ugrvJxRTYux05vqQjgsWP1ClFrlqqWjynv7sc7nQknj9zEdRRr5bz0b/7RBaASguPB2R1mK5hkISlqHeGMk6IFoC2ksczIIzPXHXHxiDG7fqEyi1OvH 2021-03-21T11:55:49.369 INFO:teuthology.kill:2021-03-21 11:55:49,369.369 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-21T11:55:49.373 INFO:teuthology.kill:2021-03-21 11:55:49,373.373 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-21T11:55:49.374 INFO:teuthology.kill:2021-03-21 11:55:49,374.374 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-21T11:55:49.379 INFO:teuthology.kill:2021-03-21 11:55:49,379.379 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-21T11:55:49.380 INFO:teuthology.kill:2021-03-21 11:55:49,380.380 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-21T11:55:49.438 INFO:teuthology.kill:2021-03-21 11:55:49,438.438 INFO:teuthology.orchestra.console:Power off gibba028 2021-03-21T11:55:49.452 INFO:teuthology.kill:2021-03-21 11:55:49,452.452 INFO:teuthology.orchestra.console:Power off gibba040 2021-03-21T11:55:49.463 INFO:teuthology.kill:2021-03-21 11:55:49,463.463 INFO:teuthology.orchestra.console:Power off gibba023 2021-03-21T11:55:49.473 INFO:teuthology.kill:2021-03-21 11:55:49,473.473 INFO:teuthology.orchestra.console:Power off gibba002 2021-03-21T11:55:49.483 INFO:teuthology.kill:2021-03-21 11:55:49,482.482 INFO:teuthology.orchestra.console:Power off gibba025 2021-03-21T11:55:49.622 INFO:teuthology.kill:2021-03-21 11:55:49,621.621 INFO:teuthology.orchestra.console:Power off for gibba040 completed 2021-03-21T11:55:49.623 INFO:teuthology.kill:2021-03-21 11:55:49,623.623 INFO:teuthology.orchestra.console:Power off for gibba023 completed 2021-03-21T11:55:49.625 INFO:teuthology.kill:2021-03-21 11:55:49,625.625 INFO:teuthology.orchestra.console:Power off for gibba028 completed 2021-03-21T11:55:49.631 INFO:teuthology.kill:2021-03-21 11:55:49,631.631 INFO:teuthology.orchestra.console:Power off for gibba002 completed 2021-03-21T11:55:49.638 INFO:teuthology.kill:2021-03-21 11:55:49,638.638 INFO:teuthology.orchestra.console:Power off for gibba025 completed 2021-03-21T11:55:49.777 INFO:teuthology.kill:2021-03-21 11:55:49,776.776 INFO:teuthology.lock.ops:unlocked gibba023.front.sepia.ceph.com 2021-03-21T11:55:49.781 INFO:teuthology.kill:2021-03-21 11:55:49,781.781 INFO:teuthology.lock.ops:unlocked gibba002.front.sepia.ceph.com 2021-03-21T11:55:49.782 INFO:teuthology.kill:2021-03-21 11:55:49,782.782 ERROR:teuthology.lock.ops:failed to unlock gibba040.front.sepia.ceph.com. reason: 500 2021-03-21T11:55:49.784 INFO:teuthology.kill:2021-03-21 11:55:49,784.784 ERROR:teuthology.lock.ops:failed to unlock gibba028.front.sepia.ceph.com. reason: 500 2021-03-21T11:55:49.797 INFO:teuthology.kill:2021-03-21 11:55:49,797.797 ERROR:teuthology.lock.ops:failed to unlock gibba025.front.sepia.ceph.com. reason: 500 2021-03-21T11:57:50.101 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-03-21T11:57:50.311 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-03-21T11:57:50.312 INFO:teuthology.nuke:Checking targets against current locks 2021-03-21T11:57:50.417 INFO:teuthology.nuke:Not nuking gibba023.front.sepia.ceph.com because description doesn't match 2021-03-21T11:57:50.418 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/scripts/dispatcher.py", line 33, in main teuthology.dispatcher.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/dispatcher/supervisor.py", line 59, in main verbose File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/dispatcher/supervisor.py", line 152, in run_job unlock_targets(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/dispatcher/supervisor.py", line 196, in unlock_targets nuke(fake_ctx, True) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6eecfe89a4363750d696160843a096384b357a8/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable