2021-02-23T03:40:40.012 INFO:root:teuthology version: 1.1.0-9a6b849c 2021-02-23T03:40:40.190 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi007.front.sepia.ceph.com' reimaging 2021-02-23T03:40:40.192 INFO:teuthology.lock.ops:Updating [ubuntu@smithi007.front.sepia.ceph.com]: reset os type and version on server 2021-02-23T03:40:40.192 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2021-02-23T03:40:40.206 INFO:teuthology.lock.ops:Node 'ubuntu@smithi007.front.sepia.ceph.com' reimaging is complete 2021-02-23T03:40:40.206 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi008.front.sepia.ceph.com' reimaging 2021-02-23T03:40:40.207 INFO:teuthology.lock.ops:Updating [ubuntu@smithi008.front.sepia.ceph.com]: reset os type and version on server 2021-02-23T03:40:40.207 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2021-02-23T03:40:40.221 INFO:teuthology.lock.ops:Node 'ubuntu@smithi008.front.sepia.ceph.com' reimaging is complete 2021-02-23T03:40:40.222 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi044.front.sepia.ceph.com' reimaging 2021-02-23T03:40:40.222 INFO:teuthology.lock.ops:Updating [ubuntu@smithi044.front.sepia.ceph.com]: reset os type and version on server 2021-02-23T03:40:40.223 INFO:teuthology.lock.ops:Updating smithi044.front.sepia.ceph.com on lock server 2021-02-23T03:40:40.239 INFO:teuthology.lock.ops:Node 'ubuntu@smithi044.front.sepia.ceph.com' reimaging is complete 2021-02-23T03:40:40.239 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi069.front.sepia.ceph.com' reimaging 2021-02-23T03:40:40.240 INFO:teuthology.lock.ops:Updating [ubuntu@smithi069.front.sepia.ceph.com]: reset os type and version on server 2021-02-23T03:40:40.240 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2021-02-23T03:40:40.256 INFO:teuthology.lock.ops:Node 'ubuntu@smithi069.front.sepia.ceph.com' reimaging is complete 2021-02-23T03:40:40.256 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi103.front.sepia.ceph.com' reimaging 2021-02-23T03:40:40.257 INFO:teuthology.lock.ops:Updating [ubuntu@smithi103.front.sepia.ceph.com]: reset os type and version on server 2021-02-23T03:40:40.257 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2021-02-23T03:40:40.280 INFO:teuthology.lock.ops:Node 'ubuntu@smithi103.front.sepia.ceph.com' reimaging is complete 2021-02-23T03:40:40.441 INFO:teuthology.provision.fog.smithi007:Scheduling deploy of ubuntu 18.04 2021-02-23T03:40:40.447 INFO:teuthology.provision.fog.smithi044:Scheduling deploy of ubuntu 18.04 2021-02-23T03:40:40.461 INFO:teuthology.provision.fog.smithi008:Scheduling deploy of ubuntu 18.04 2021-02-23T03:40:40.492 INFO:teuthology.provision.fog.smithi069:Scheduling deploy of ubuntu 18.04 2021-02-23T03:40:40.526 INFO:teuthology.provision.fog.smithi103:Scheduling deploy of ubuntu 18.04 2021-02-23T03:40:40.790 INFO:teuthology.orchestra.console:Power off smithi007 2021-02-23T03:40:40.807 INFO:teuthology.orchestra.console:Power off smithi008 2021-02-23T03:40:40.844 INFO:teuthology.orchestra.console:Power off smithi044 2021-02-23T03:40:40.875 INFO:teuthology.orchestra.console:Power off smithi069 2021-02-23T03:40:40.886 INFO:teuthology.orchestra.console:Power off smithi103 2021-02-23T03:40:41.016 INFO:teuthology.orchestra.console:Power off for smithi069 completed 2021-02-23T03:40:41.025 INFO:teuthology.orchestra.console:Power off for smithi103 completed 2021-02-23T03:40:41.117 INFO:teuthology.orchestra.console:Power on smithi069 2021-02-23T03:40:41.127 INFO:teuthology.orchestra.console:Power on smithi103 2021-02-23T03:40:49.174 INFO:teuthology.orchestra.console:Power off for smithi007 completed 2021-02-23T03:40:49.184 INFO:teuthology.orchestra.console:Power off for smithi008 completed 2021-02-23T03:40:49.216 INFO:teuthology.orchestra.console:Power off for smithi044 completed 2021-02-23T03:40:49.277 INFO:teuthology.orchestra.console:Power on smithi007 2021-02-23T03:40:49.294 INFO:teuthology.orchestra.console:Power on smithi008 2021-02-23T03:40:49.320 INFO:teuthology.orchestra.console:Power on smithi044 2021-02-23T03:40:49.479 INFO:teuthology.orchestra.console:Power on for smithi069 completed 2021-02-23T03:40:49.496 INFO:teuthology.orchestra.console:Power on for smithi103 completed 2021-02-23T03:40:49.581 INFO:teuthology.provision.fog.smithi069:Waiting for deploy to finish 2021-02-23T03:40:49.597 INFO:teuthology.provision.fog.smithi103:Waiting for deploy to finish 2021-02-23T03:40:57.652 INFO:teuthology.orchestra.console:Power on for smithi007 completed 2021-02-23T03:40:57.658 INFO:teuthology.orchestra.console:Power on for smithi008 completed 2021-02-23T03:40:57.680 INFO:teuthology.orchestra.console:Power on for smithi044 completed 2021-02-23T03:40:57.753 INFO:teuthology.provision.fog.smithi007:Waiting for deploy to finish 2021-02-23T03:40:57.759 INFO:teuthology.provision.fog.smithi008:Waiting for deploy to finish 2021-02-23T03:40:57.782 INFO:teuthology.provision.fog.smithi044:Waiting for deploy to finish 2021-02-23T03:43:50.837 ERROR:teuthology.orchestra.connection:Error authenticating with smithi103.front.sepia.ceph.com: Authentication failed. 2021-02-23T03:45:00.615 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-23T03:45:00.616 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-23T03:45:06.628 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-23T03:45:06.629 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-23T03:45:12.640 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-23T03:45:12.641 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-23T03:45:15.832 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-23T03:45:19.660 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-23T03:45:21.845 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-23T03:45:24.733 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-23T03:45:25.133 INFO:teuthology.provision.fog.smithi044:Node is ready 2021-02-23T03:45:25.181 INFO:teuthology.orchestra.run.smithi044.stdout:smithi044.front.sepia.ceph.com 2021-02-23T03:45:25.230 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44 smithi044.front.sepia.ceph.com smithi044 2021-02-23T03:45:25.560 INFO:teuthology.provision.fog.smithi044:Deploy complete! 2021-02-23T03:45:25.738 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-23T03:45:26.240 INFO:teuthology.provision.fog.smithi007:Node is ready 2021-02-23T03:45:26.246 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007.front.sepia.ceph.com 2021-02-23T03:45:26.296 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7 smithi007.front.sepia.ceph.com smithi007 2021-02-23T03:45:26.574 INFO:teuthology.provision.fog.smithi007:Deploy complete! 2021-02-23T03:45:27.857 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-23T03:45:34.892 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-23T03:45:40.970 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-23T03:45:41.479 INFO:teuthology.provision.fog.smithi008:Node is ready 2021-02-23T03:45:41.486 INFO:teuthology.orchestra.run.smithi008.stdout:smithi008.front.sepia.ceph.com 2021-02-23T03:45:41.532 INFO:teuthology.orchestra.run.smithi008.stdout:172.21.15.8 smithi008.front.sepia.ceph.com smithi008 2021-02-23T03:45:41.843 INFO:teuthology.provision.fog.smithi008:Deploy complete! 2021-02-23T03:45:42.129 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-23T03:45:42.509 INFO:teuthology.provision.fog.smithi069:Node is ready 2021-02-23T03:45:42.557 INFO:teuthology.orchestra.run.smithi069.stdout:smithi069.front.sepia.ceph.com 2021-02-23T03:45:42.606 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69 smithi069.front.sepia.ceph.com smithi069 2021-02-23T03:45:42.901 INFO:teuthology.provision.fog.smithi069:Deploy complete! 2021-02-23T03:46:00.658 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-23T03:46:07.017 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-23T03:46:07.513 INFO:teuthology.provision.fog.smithi103:Node is ready 2021-02-23T03:46:07.519 INFO:teuthology.orchestra.run.smithi103.stdout:smithi103.front.sepia.ceph.com 2021-02-23T03:46:07.566 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103 smithi103.front.sepia.ceph.com smithi103 2021-02-23T03:46:07.850 INFO:teuthology.provision.fog.smithi103:Deploy complete! 2021-02-23T03:46:08.082 INFO:teuthology.lock.ops:Checking smithi007.front.sepia.ceph.com 2021-02-23T03:46:08.082 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-23T03:46:08.098 INFO:teuthology.lock.ops:Checking smithi008.front.sepia.ceph.com 2021-02-23T03:46:08.099 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-23T03:46:08.111 INFO:teuthology.lock.ops:Checking smithi044.front.sepia.ceph.com 2021-02-23T03:46:08.111 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-23T03:46:08.125 INFO:teuthology.lock.ops:Checking smithi069.front.sepia.ceph.com 2021-02-23T03:46:08.126 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-23T03:46:08.137 INFO:teuthology.lock.ops:Checking smithi103.front.sepia.ceph.com 2021-02-23T03:46:08.138 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-23T03:46:08.150 INFO:teuthology.lock.ops:Updating [smithi007.front.sepia.ceph.com]: set os type and version on server 2021-02-23T03:46:08.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-23T03:46:08.509 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2021-02-23T03:46:08.557 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="Ubuntu" 2021-02-23T03:46:08.558 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-23T03:46:08.558 INFO:teuthology.orchestra.run.smithi007.stdout:ID=ubuntu 2021-02-23T03:46:08.558 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE=debian 2021-02-23T03:46:08.558 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-23T03:46:08.558 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="18.04" 2021-02-23T03:46:08.559 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-23T03:46:08.559 INFO:teuthology.orchestra.run.smithi007.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-23T03:46:08.559 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-23T03:46:08.559 INFO:teuthology.orchestra.run.smithi007.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-23T03:46:08.559 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_CODENAME=bionic 2021-02-23T03:46:08.560 INFO:teuthology.orchestra.run.smithi007.stdout:UBUNTU_CODENAME=bionic 2021-02-23T03:46:08.561 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2021-02-23T03:46:08.576 INFO:teuthology.lock.ops:Updating [smithi008.front.sepia.ceph.com]: set os type and version on server 2021-02-23T03:46:08.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-23T03:46:08.942 INFO:teuthology.orchestra.run.smithi008.stdout:x86_64 2021-02-23T03:46:08.990 INFO:teuthology.orchestra.run.smithi008.stdout:NAME="Ubuntu" 2021-02-23T03:46:08.990 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-23T03:46:08.990 INFO:teuthology.orchestra.run.smithi008.stdout:ID=ubuntu 2021-02-23T03:46:08.991 INFO:teuthology.orchestra.run.smithi008.stdout:ID_LIKE=debian 2021-02-23T03:46:08.991 INFO:teuthology.orchestra.run.smithi008.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-23T03:46:08.991 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_ID="18.04" 2021-02-23T03:46:08.991 INFO:teuthology.orchestra.run.smithi008.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-23T03:46:08.991 INFO:teuthology.orchestra.run.smithi008.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-23T03:46:08.992 INFO:teuthology.orchestra.run.smithi008.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-23T03:46:08.992 INFO:teuthology.orchestra.run.smithi008.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-23T03:46:08.992 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_CODENAME=bionic 2021-02-23T03:46:08.992 INFO:teuthology.orchestra.run.smithi008.stdout:UBUNTU_CODENAME=bionic 2021-02-23T03:46:08.993 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2021-02-23T03:46:09.008 INFO:teuthology.lock.ops:Updating [smithi044.front.sepia.ceph.com]: set os type and version on server 2021-02-23T03:46:09.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-23T03:46:09.338 INFO:teuthology.orchestra.run.smithi044.stdout:x86_64 2021-02-23T03:46:09.450 INFO:teuthology.orchestra.run.smithi044.stdout:NAME="Ubuntu" 2021-02-23T03:46:09.452 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-23T03:46:09.454 INFO:teuthology.orchestra.run.smithi044.stdout:ID=ubuntu 2021-02-23T03:46:09.454 INFO:teuthology.orchestra.run.smithi044.stdout:ID_LIKE=debian 2021-02-23T03:46:09.457 INFO:teuthology.orchestra.run.smithi044.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-23T03:46:09.457 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION_ID="18.04" 2021-02-23T03:46:09.457 INFO:teuthology.orchestra.run.smithi044.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-23T03:46:09.458 INFO:teuthology.orchestra.run.smithi044.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-23T03:46:09.458 INFO:teuthology.orchestra.run.smithi044.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-23T03:46:09.458 INFO:teuthology.orchestra.run.smithi044.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-23T03:46:09.459 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION_CODENAME=bionic 2021-02-23T03:46:09.459 INFO:teuthology.orchestra.run.smithi044.stdout:UBUNTU_CODENAME=bionic 2021-02-23T03:46:09.460 INFO:teuthology.lock.ops:Updating smithi044.front.sepia.ceph.com on lock server 2021-02-23T03:46:09.479 INFO:teuthology.lock.ops:Updating [smithi069.front.sepia.ceph.com]: set os type and version on server 2021-02-23T03:46:09.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-23T03:46:09.899 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2021-02-23T03:46:09.955 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="Ubuntu" 2021-02-23T03:46:09.956 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-23T03:46:09.956 INFO:teuthology.orchestra.run.smithi069.stdout:ID=ubuntu 2021-02-23T03:46:09.957 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE=debian 2021-02-23T03:46:09.958 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-23T03:46:09.959 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="18.04" 2021-02-23T03:46:09.959 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-23T03:46:09.959 INFO:teuthology.orchestra.run.smithi069.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-23T03:46:09.959 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-23T03:46:09.960 INFO:teuthology.orchestra.run.smithi069.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-23T03:46:09.960 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_CODENAME=bionic 2021-02-23T03:46:09.960 INFO:teuthology.orchestra.run.smithi069.stdout:UBUNTU_CODENAME=bionic 2021-02-23T03:46:09.961 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2021-02-23T03:46:09.978 INFO:teuthology.lock.ops:Updating [smithi103.front.sepia.ceph.com]: set os type and version on server 2021-02-23T03:46:09.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-23T03:46:10.318 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2021-02-23T03:46:10.366 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="Ubuntu" 2021-02-23T03:46:10.367 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-02-23T03:46:10.367 INFO:teuthology.orchestra.run.smithi103.stdout:ID=ubuntu 2021-02-23T03:46:10.367 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE=debian 2021-02-23T03:46:10.367 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-02-23T03:46:10.368 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="18.04" 2021-02-23T03:46:10.368 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://www.ubuntu.com/" 2021-02-23T03:46:10.368 INFO:teuthology.orchestra.run.smithi103.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-02-23T03:46:10.368 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-02-23T03:46:10.368 INFO:teuthology.orchestra.run.smithi103.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-02-23T03:46:10.369 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_CODENAME=bionic 2021-02-23T03:46:10.369 INFO:teuthology.orchestra.run.smithi103.stdout:UBUNTU_CODENAME=bionic 2021-02-23T03:46:10.369 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2021-02-23T03:46:10.476 INFO:teuthology.dispatcher.supervisor:Running job 5905609 2021-02-23T03:46:10.477 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-02-23_02:30:05-upgrade:mimic-x-nautilus-distro-basic-smithi/5905609 --name teuthology-2021-02-23_02:30:05-upgrade:mimic-x-nautilus-distro-basic-smithi --description upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-nautilus 6.1-msgr2 7-final-workload/{rbd-python rgw-swift-ragweed_check snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_latest thrashosds-health} -- /home/teuthworker/archive/teuthology-2021-02-23_02:30:05-upgrade:mimic-x-nautilus-distro-basic-smithi/5905609/orig.config.yaml 2021-02-23T03:46:10.480 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-02-23_02:30:05-upgrade:mimic-x-nautilus-distro-basic-smithi/5905609 2021-02-23T03:46:10.482 INFO:teuthology.dispatcher.supervisor:Job PID: 8364 2021-02-23T03:46:10.482 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-02-23T15:46:32.356 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-02-23T15:46:41.556 INFO:teuthology.kill:Killing Pids: {8364} 2021-02-23T15:46:43.681 INFO:teuthology.kill:Nuking machines: ['smithi007', 'smithi008', 'smithi044', 'smithi069', 'smithi103'] 2021-02-23T15:46:45.155 INFO:teuthology.kill:2021-02-23 15:46:45,155.155 INFO:teuthology.nuke:targets: 2021-02-23T15:46:45.158 INFO:teuthology.kill: smithi007.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDycB1Tcl20vHHKSPcvnQhc4sTojDHtq3IAIdwQla/mtf5JJf2DrAMzzYgrUd+MzvRLx3mv74lz+nHCa+NS0/NOoSklHJIZWatWB2Hjc79ImrLQwxjupYWh/a1VDUG9S+D896nVaaXWbT/mfraRIHP5TIsNwcUmtUQYPlLA9eI0SrWYeRAUja6DQDwmrlKb7ajNkEdHq/36joWqVa3fEdihucfwr/wiRDLVN8x/fgpG3ZwRMPV6S7X0oVYDfFt+vmJ3OrnzU6BaCTwTs4x8y10ab1nDHeuJlRV6sbjhrL25PNRuXAu65S3owIQANrtd9gRU5r+/TzzOFByTKR7ZQo8F 2021-02-23T15:46:45.159 INFO:teuthology.kill: smithi008.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZRPahDzg/cUkrE/QCJKl3dK3nxiSLJfO2XFBQD2wRqLLRxzuFShbUsvru/RAEOCWquAAjgTMW3kNwadGivhzeDCOKGqMghPRpB7nswweD8a/0PKrzP+JAzVizGA2duv/cb/BQtQG7SX4rrY5IygzW01QTyUPMpp0+6fZ6FKk9iX9zEO2NWZcd4iRj7cuomrd8PgMJ6I8suT33agh1yMQHoSS67AlEucLY4jrYph7g//8zeFiPu4I3bnCO1nARu3ywvUFAvuqiHT/TgSkOtSd19LLS36ETaGX8t66IskuxzsWloXn0yfL/IAC2l9is+guEWkPE4wLlT/esSXZkH/NL 2021-02-23T15:46:45.159 INFO:teuthology.kill: smithi044.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCmoqw4Ic5UD0VtkawtY9GiKnHO1lBRIhV7LG9vHaFICnXbE24JOZYjgNPaJ2sOv/hWJAsdJ6SMp+4qooIu9r/h9iGzTjR32FRx2Hh+HddjlO5zyLCLO+rxislWw9tQL0dIB4cxMVn6r5/ZN7mx152SvxvNJnzRU3tOA+VBY8IBIljbpQJXmBv7B/0AkqQvrHcAQbK+LHlZCN4zLTfNIj0TQ8RDU9qpF5ASB4m+GqbKKpNqTudzIEG8xL0Nk0iDZy8FVUy+BFBkIdjtjsFvrM+ioZGvB5HF3omqYyNu9fMF3F/bn06D3RTE4LsAMtccwZaL4ScodVYKDAJPzV9YFVC3 2021-02-23T15:46:45.160 INFO:teuthology.kill: smithi069.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn+UcQm+n1sfHgIz6bf6mCb7xz08r/6b/FmGYppDH2eRiJ25qtIAn5ta9j0oheP1zgfVdgaqNNScUEAfd51AI5z25Gklb7YsejWjIwQ+HJTp3HhJjRFt61J1waOj4l7yBoFNikSbw7sRbFMk3rS6nZKLU0yZ8SnDNLr3KjGq0OgWjkhlaA58Grdw2mvXPPFnEQo+3JyL+5+QX7063RJSoGKTrOKC3kBXeuKmxrze+P8+zh9E20Gp1BTkh9WTIlwFObVaKWUJBlMvfStNLkzF1q9e5sLRn43mVc76zm2fzAVZYWlv5YnYf0yJBsca0T2edBE4Bkq8HNpIGf6HpHglnP 2021-02-23T15:46:45.160 INFO:teuthology.kill: smithi103.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUr+6BY030r3/wmG3ltcG8y7neFstgnNuhD4J+5ioBfNTSVNNCd7Y67/WKYfIZF/HNEmF5crrl42DfnYqkNTr1+bFH7RaLemYCaIMKWeiO4WqFmZE2tMfUCQ2IFRyJnyBZO7FRT8kb046iTGXhcfFc8i62G3QAXJNu92GrIrBTlyip79sZc1r6ArCtYSvgUvYWnzXVufLv2OH8s9wvbGuEX9dGUCy36RiqlTPUz0JgJZADbk2yYeIOjaSkjpeTpqj/yRkxcqD2Bx8lDDBZsN8AtLCdcbtPdvFb3fAsDDCLlUc0Bd1wEAVxivr2RKz+u78pBlq+iweY8tVFmevE71Bd 2021-02-23T15:46:45.160 INFO:teuthology.kill:2021-02-23 15:46:45,155.155 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:46:45.217 INFO:teuthology.kill:2021-02-23 15:46:45,217.217 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:46:45.223 INFO:teuthology.kill:2021-02-23 15:46:45,221.221 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:46:45.228 INFO:teuthology.kill:2021-02-23 15:46:45,227.227 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:46:45.231 INFO:teuthology.kill:2021-02-23 15:46:45,230.230 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:46:45.352 INFO:teuthology.kill:2021-02-23 15:46:45,351.351 INFO:teuthology.orchestra.console:Power off smithi103 2021-02-23T15:46:45.399 INFO:teuthology.kill:2021-02-23 15:46:45,398.398 INFO:teuthology.orchestra.console:Power off smithi008 2021-02-23T15:46:45.424 INFO:teuthology.kill:2021-02-23 15:46:45,422.422 INFO:teuthology.orchestra.console:Power off smithi044 2021-02-23T15:46:45.440 INFO:teuthology.kill:2021-02-23 15:46:45,440.440 INFO:teuthology.orchestra.console:Power off smithi007 2021-02-23T15:46:45.463 INFO:teuthology.kill:2021-02-23 15:46:45,463.463 INFO:teuthology.orchestra.console:Power off smithi069 2021-02-23T15:46:53.854 INFO:teuthology.kill:2021-02-23 15:46:53,854.854 INFO:teuthology.orchestra.console:Power off for smithi008 completed 2021-02-23T15:46:53.882 INFO:teuthology.kill:2021-02-23 15:46:53,882.882 INFO:teuthology.orchestra.console:Power off for smithi103 completed 2021-02-23T15:46:57.977 INFO:teuthology.kill:2021-02-23 15:46:57,977.977 INFO:teuthology.orchestra.console:Power off for smithi044 completed 2021-02-23T15:46:57.987 INFO:teuthology.kill:2021-02-23 15:46:57,987.987 INFO:teuthology.orchestra.console:Power off for smithi007 completed 2021-02-23T15:46:58.001 INFO:teuthology.kill:2021-02-23 15:46:58,001.001 INFO:teuthology.orchestra.console:Power off for smithi069 completed 2021-02-23T15:46:58.361 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2021-02-23T15:46:58.369 INFO:teuthology.task.internal:roles: ubuntu@smithi008.front.sepia.ceph.com - ['mon.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2021-02-23T15:46:58.370 INFO:teuthology.task.internal:roles: ubuntu@smithi044.front.sepia.ceph.com - ['mon.c'] 2021-02-23T15:46:58.370 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11'] 2021-02-23T15:46:58.371 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['client.0'] 2021-02-23T15:46:58.374 INFO:teuthology.misc:Compressing logs... 2021-02-23T15:46:58.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-23T15:47:33.201 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-02-23T15:47:36.027 INFO:teuthology.kill:No teuthology processes running 2021-02-23T15:47:37.036 INFO:teuthology.kill:Nuking machines: ['smithi007', 'smithi008', 'smithi044', 'smithi069', 'smithi103'] 2021-02-23T15:47:38.131 INFO:teuthology.kill:2021-02-23 15:47:38,131.131 INFO:teuthology.nuke:targets: 2021-02-23T15:47:38.132 INFO:teuthology.kill: smithi007.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDycB1Tcl20vHHKSPcvnQhc4sTojDHtq3IAIdwQla/mtf5JJf2DrAMzzYgrUd+MzvRLx3mv74lz+nHCa+NS0/NOoSklHJIZWatWB2Hjc79ImrLQwxjupYWh/a1VDUG9S+D896nVaaXWbT/mfraRIHP5TIsNwcUmtUQYPlLA9eI0SrWYeRAUja6DQDwmrlKb7ajNkEdHq/36joWqVa3fEdihucfwr/wiRDLVN8x/fgpG3ZwRMPV6S7X0oVYDfFt+vmJ3OrnzU6BaCTwTs4x8y10ab1nDHeuJlRV6sbjhrL25PNRuXAu65S3owIQANrtd9gRU5r+/TzzOFByTKR7ZQo8F 2021-02-23T15:47:38.132 INFO:teuthology.kill: smithi008.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZRPahDzg/cUkrE/QCJKl3dK3nxiSLJfO2XFBQD2wRqLLRxzuFShbUsvru/RAEOCWquAAjgTMW3kNwadGivhzeDCOKGqMghPRpB7nswweD8a/0PKrzP+JAzVizGA2duv/cb/BQtQG7SX4rrY5IygzW01QTyUPMpp0+6fZ6FKk9iX9zEO2NWZcd4iRj7cuomrd8PgMJ6I8suT33agh1yMQHoSS67AlEucLY4jrYph7g//8zeFiPu4I3bnCO1nARu3ywvUFAvuqiHT/TgSkOtSd19LLS36ETaGX8t66IskuxzsWloXn0yfL/IAC2l9is+guEWkPE4wLlT/esSXZkH/NL 2021-02-23T15:47:38.133 INFO:teuthology.kill: smithi044.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCmoqw4Ic5UD0VtkawtY9GiKnHO1lBRIhV7LG9vHaFICnXbE24JOZYjgNPaJ2sOv/hWJAsdJ6SMp+4qooIu9r/h9iGzTjR32FRx2Hh+HddjlO5zyLCLO+rxislWw9tQL0dIB4cxMVn6r5/ZN7mx152SvxvNJnzRU3tOA+VBY8IBIljbpQJXmBv7B/0AkqQvrHcAQbK+LHlZCN4zLTfNIj0TQ8RDU9qpF5ASB4m+GqbKKpNqTudzIEG8xL0Nk0iDZy8FVUy+BFBkIdjtjsFvrM+ioZGvB5HF3omqYyNu9fMF3F/bn06D3RTE4LsAMtccwZaL4ScodVYKDAJPzV9YFVC3 2021-02-23T15:47:38.133 INFO:teuthology.kill: smithi069.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn+UcQm+n1sfHgIz6bf6mCb7xz08r/6b/FmGYppDH2eRiJ25qtIAn5ta9j0oheP1zgfVdgaqNNScUEAfd51AI5z25Gklb7YsejWjIwQ+HJTp3HhJjRFt61J1waOj4l7yBoFNikSbw7sRbFMk3rS6nZKLU0yZ8SnDNLr3KjGq0OgWjkhlaA58Grdw2mvXPPFnEQo+3JyL+5+QX7063RJSoGKTrOKC3kBXeuKmxrze+P8+zh9E20Gp1BTkh9WTIlwFObVaKWUJBlMvfStNLkzF1q9e5sLRn43mVc76zm2fzAVZYWlv5YnYf0yJBsca0T2edBE4Bkq8HNpIGf6HpHglnP 2021-02-23T15:47:38.133 INFO:teuthology.kill: smithi103.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUr+6BY030r3/wmG3ltcG8y7neFstgnNuhD4J+5ioBfNTSVNNCd7Y67/WKYfIZF/HNEmF5crrl42DfnYqkNTr1+bFH7RaLemYCaIMKWeiO4WqFmZE2tMfUCQ2IFRyJnyBZO7FRT8kb046iTGXhcfFc8i62G3QAXJNu92GrIrBTlyip79sZc1r6ArCtYSvgUvYWnzXVufLv2OH8s9wvbGuEX9dGUCy36RiqlTPUz0JgJZADbk2yYeIOjaSkjpeTpqj/yRkxcqD2Bx8lDDBZsN8AtLCdcbtPdvFb3fAsDDCLlUc0Bd1wEAVxivr2RKz+u78pBlq+iweY8tVFmevE71Bd 2021-02-23T15:47:38.164 INFO:teuthology.kill:2021-02-23 15:47:38,164.164 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:47:38.167 INFO:teuthology.kill:2021-02-23 15:47:38,167.167 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:47:38.171 INFO:teuthology.kill:2021-02-23 15:47:38,170.170 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:47:38.174 INFO:teuthology.kill:2021-02-23 15:47:38,173.173 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:47:38.176 INFO:teuthology.kill:2021-02-23 15:47:38,176.176 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:47:38.229 INFO:teuthology.kill:2021-02-23 15:47:38,228.228 INFO:teuthology.orchestra.console:Power off smithi044 2021-02-23T15:47:38.242 INFO:teuthology.kill:2021-02-23 15:47:38,242.242 INFO:teuthology.orchestra.console:Power off smithi069 2021-02-23T15:47:38.256 INFO:teuthology.kill:2021-02-23 15:47:38,255.255 INFO:teuthology.orchestra.console:Power off smithi007 2021-02-23T15:47:38.272 INFO:teuthology.kill:2021-02-23 15:47:38,271.271 INFO:teuthology.orchestra.console:Power off smithi103 2021-02-23T15:47:38.286 INFO:teuthology.kill:2021-02-23 15:47:38,285.285 INFO:teuthology.orchestra.console:Power off smithi008 2021-02-23T15:47:38.426 INFO:teuthology.kill:2021-02-23 15:47:38,426.426 INFO:teuthology.orchestra.console:Power off for smithi069 completed 2021-02-23T15:47:38.428 INFO:teuthology.kill:2021-02-23 15:47:38,428.428 INFO:teuthology.orchestra.console:Power off for smithi007 completed 2021-02-23T15:47:38.428 INFO:teuthology.kill:2021-02-23 15:47:38,428.428 INFO:teuthology.orchestra.console:Power off for smithi044 completed 2021-02-23T15:47:38.433 INFO:teuthology.kill:2021-02-23 15:47:38,433.433 INFO:teuthology.orchestra.console:Power off for smithi103 completed 2021-02-23T15:47:38.439 INFO:teuthology.kill:2021-02-23 15:47:38,439.439 INFO:teuthology.orchestra.console:Power off for smithi008 completed 2021-02-23T15:47:38.580 INFO:teuthology.kill:2021-02-23 15:47:38,579.579 INFO:teuthology.lock.ops:unlocked smithi069.front.sepia.ceph.com 2021-02-23T15:47:38.583 INFO:teuthology.kill:2021-02-23 15:47:38,583.583 INFO:teuthology.lock.ops:unlocked smithi008.front.sepia.ceph.com 2021-02-23T15:47:38.584 INFO:teuthology.kill:2021-02-23 15:47:38,584.584 INFO:teuthology.lock.ops:unlocked smithi007.front.sepia.ceph.com 2021-02-23T15:47:38.585 INFO:teuthology.kill:2021-02-23 15:47:38,585.585 INFO:teuthology.lock.ops:unlocked smithi044.front.sepia.ceph.com 2021-02-23T15:47:38.588 INFO:teuthology.kill:2021-02-23 15:47:38,587.587 INFO:teuthology.lock.ops:unlocked smithi103.front.sepia.ceph.com 2021-02-23T15:49:38.788 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-02-23T15:49:40.103 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-02-23T15:49:40.103 INFO:teuthology.nuke:Checking targets against current locks 2021-02-23T15:49:40.519 INFO:teuthology.nuke:Not nuking smithi008.front.sepia.ceph.com because description doesn't match 2021-02-23T15:49:40.542 INFO:teuthology.nuke:Not nuking smithi069.front.sepia.ceph.com because description doesn't match 2021-02-23T15:49:40.542 INFO:teuthology.nuke:Not nuking smithi044.front.sepia.ceph.com because description doesn't match 2021-02-23T15:49:40.543 INFO:teuthology.nuke:Not nuking smithi103.front.sepia.ceph.com because description doesn't match 2021-02-23T15:49:40.543 INFO:teuthology.nuke:Not nuking smithi007.front.sepia.ceph.com because description doesn't match