2021-02-23T03:51:23.087 INFO:root:teuthology version: 1.1.0-9a6b849c 2021-02-23T03:51:23.287 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi023.front.sepia.ceph.com' reimaging 2021-02-23T03:51:23.288 INFO:teuthology.lock.ops:Updating [ubuntu@smithi023.front.sepia.ceph.com]: reset os type and version on server 2021-02-23T03:51:23.288 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2021-02-23T03:51:23.307 INFO:teuthology.lock.ops:Node 'ubuntu@smithi023.front.sepia.ceph.com' reimaging is complete 2021-02-23T03:51:23.308 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi041.front.sepia.ceph.com' reimaging 2021-02-23T03:51:23.308 INFO:teuthology.lock.ops:Updating [ubuntu@smithi041.front.sepia.ceph.com]: reset os type and version on server 2021-02-23T03:51:23.309 INFO:teuthology.lock.ops:Updating smithi041.front.sepia.ceph.com on lock server 2021-02-23T03:51:23.327 INFO:teuthology.lock.ops:Node 'ubuntu@smithi041.front.sepia.ceph.com' reimaging is complete 2021-02-23T03:51:23.328 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi066.front.sepia.ceph.com' reimaging 2021-02-23T03:51:23.328 INFO:teuthology.lock.ops:Updating [ubuntu@smithi066.front.sepia.ceph.com]: reset os type and version on server 2021-02-23T03:51:23.329 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server 2021-02-23T03:51:23.345 INFO:teuthology.lock.ops:Node 'ubuntu@smithi066.front.sepia.ceph.com' reimaging is complete 2021-02-23T03:51:23.346 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi148.front.sepia.ceph.com' reimaging 2021-02-23T03:51:23.347 INFO:teuthology.lock.ops:Updating [ubuntu@smithi148.front.sepia.ceph.com]: reset os type and version on server 2021-02-23T03:51:23.347 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2021-02-23T03:51:23.365 INFO:teuthology.lock.ops:Node 'ubuntu@smithi148.front.sepia.ceph.com' reimaging is complete 2021-02-23T03:51:23.365 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi178.front.sepia.ceph.com' reimaging 2021-02-23T03:51:23.366 INFO:teuthology.lock.ops:Updating [ubuntu@smithi178.front.sepia.ceph.com]: reset os type and version on server 2021-02-23T03:51:23.366 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2021-02-23T03:51:23.383 INFO:teuthology.lock.ops:Node 'ubuntu@smithi178.front.sepia.ceph.com' reimaging is complete 2021-02-23T03:51:23.521 INFO:teuthology.provision.fog.smithi023:Scheduling deploy of centos 7.8 2021-02-23T03:51:23.527 INFO:teuthology.provision.fog.smithi041:Scheduling deploy of centos 7.8 2021-02-23T03:51:23.553 INFO:teuthology.provision.fog.smithi066:Scheduling deploy of centos 7.8 2021-02-23T03:51:23.579 INFO:teuthology.provision.fog.smithi148:Scheduling deploy of centos 7.8 2021-02-23T03:51:23.586 INFO:teuthology.provision.fog.smithi178:Scheduling deploy of centos 7.8 2021-02-23T03:51:23.867 INFO:teuthology.orchestra.console:Power off smithi023 2021-02-23T03:51:23.921 INFO:teuthology.orchestra.console:Power off smithi041 2021-02-23T03:51:23.934 INFO:teuthology.orchestra.console:Power off smithi066 2021-02-23T03:51:23.980 INFO:teuthology.orchestra.console:Power off smithi178 2021-02-23T03:51:23.992 INFO:teuthology.orchestra.console:Power off smithi148 2021-02-23T03:51:32.276 INFO:teuthology.orchestra.console:Power off for smithi023 completed 2021-02-23T03:51:32.324 INFO:teuthology.orchestra.console:Power off for smithi066 completed 2021-02-23T03:51:32.331 INFO:teuthology.orchestra.console:Power off for smithi041 completed 2021-02-23T03:51:32.378 INFO:teuthology.orchestra.console:Power on smithi023 2021-02-23T03:51:32.425 INFO:teuthology.orchestra.console:Power on smithi066 2021-02-23T03:51:32.435 INFO:teuthology.orchestra.console:Power on smithi041 2021-02-23T03:51:36.457 INFO:teuthology.orchestra.console:Power off for smithi178 completed 2021-02-23T03:51:36.468 INFO:teuthology.orchestra.console:Power off for smithi148 completed 2021-02-23T03:51:36.559 INFO:teuthology.orchestra.console:Power on smithi178 2021-02-23T03:51:36.569 INFO:teuthology.orchestra.console:Power on smithi148 2021-02-23T03:51:36.632 INFO:teuthology.orchestra.console:Power on for smithi023 completed 2021-02-23T03:51:36.697 INFO:teuthology.orchestra.console:Power on for smithi041 completed 2021-02-23T03:51:36.734 INFO:teuthology.provision.fog.smithi023:Waiting for deploy to finish 2021-02-23T03:51:36.798 INFO:teuthology.provision.fog.smithi041:Waiting for deploy to finish 2021-02-23T03:51:40.798 INFO:teuthology.orchestra.console:Power on for smithi066 completed 2021-02-23T03:51:40.812 INFO:teuthology.orchestra.console:Power on for smithi178 completed 2021-02-23T03:51:40.901 INFO:teuthology.provision.fog.smithi066:Waiting for deploy to finish 2021-02-23T03:51:40.912 INFO:teuthology.provision.fog.smithi178:Waiting for deploy to finish 2021-02-23T03:51:44.936 INFO:teuthology.orchestra.console:Power on for smithi148 completed 2021-02-23T03:51:45.037 INFO:teuthology.provision.fog.smithi148:Waiting for deploy to finish 2021-02-23T03:54:12.113 ERROR:teuthology.orchestra.connection:Error authenticating with smithi066.front.sepia.ceph.com: Authentication failed. 2021-02-23T03:54:23.062 ERROR:teuthology.orchestra.connection:Error authenticating with smithi023.front.sepia.ceph.com: Authentication failed. 2021-02-23T03:55:17.821 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-23T03:55:33.636 INFO:teuthology.provision.fog.smithi041:Node is ready 2021-02-23T03:55:33.768 INFO:teuthology.orchestra.run.smithi041.stdout:smithi061 2021-02-23T03:55:33.848 INFO:teuthology.orchestra.run.smithi041.stdout:172.21.15.61 smithi061.front.sepia.ceph.com smithi061 2021-02-23T03:55:34.475 INFO:teuthology.provision.fog.smithi041:Deploy complete! 2021-02-23T03:55:56.801 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-23T03:56:06.563 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-23T03:56:17.299 INFO:teuthology.provision.fog.smithi066:Node is ready 2021-02-23T03:56:17.466 INFO:teuthology.orchestra.run.smithi066.stdout:smithi066.front.sepia.ceph.com 2021-02-23T03:56:17.558 INFO:teuthology.orchestra.run.smithi066.stdout:172.21.15.66 smithi066.front.sepia.ceph.com smithi066 2021-02-23T03:56:18.197 INFO:teuthology.provision.fog.smithi066:Deploy complete! 2021-02-23T03:56:25.421 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-23T03:56:26.927 INFO:teuthology.provision.fog.smithi148:Node is ready 2021-02-23T03:56:27.223 INFO:teuthology.orchestra.run.smithi148.stdout:smithi061 2021-02-23T03:56:27.306 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.61 smithi061.front.sepia.ceph.com smithi061 2021-02-23T03:56:27.843 INFO:teuthology.provision.fog.smithi148:Deploy complete! 2021-02-23T03:56:45.961 INFO:teuthology.provision.fog.smithi178:Node is ready 2021-02-23T03:56:46.119 INFO:teuthology.orchestra.run.smithi178.stdout:smithi178.front.sepia.ceph.com 2021-02-23T03:56:46.199 INFO:teuthology.orchestra.run.smithi178.stdout:172.21.15.178 smithi178.front.sepia.ceph.com smithi178 2021-02-23T03:56:46.637 INFO:teuthology.provision.fog.smithi178:Deploy complete! 2021-02-23T03:56:54.434 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-23T03:56:55.038 INFO:teuthology.provision.fog.smithi023:Node is ready 2021-02-23T03:56:55.128 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023.front.sepia.ceph.com 2021-02-23T03:56:55.207 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23 smithi023.front.sepia.ceph.com smithi023 2021-02-23T03:56:55.791 INFO:teuthology.provision.fog.smithi023:Deploy complete! 2021-02-23T03:56:56.106 INFO:teuthology.lock.ops:Checking smithi023.front.sepia.ceph.com 2021-02-23T03:56:56.111 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-23T03:56:56.127 INFO:teuthology.lock.ops:Checking smithi041.front.sepia.ceph.com 2021-02-23T03:56:56.128 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-23T03:56:56.140 INFO:teuthology.lock.ops:Checking smithi066.front.sepia.ceph.com 2021-02-23T03:56:56.140 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-23T03:56:56.155 INFO:teuthology.lock.ops:Checking smithi148.front.sepia.ceph.com 2021-02-23T03:56:56.156 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-23T03:56:56.170 INFO:teuthology.lock.ops:Checking smithi178.front.sepia.ceph.com 2021-02-23T03:56:56.170 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-23T03:56:56.184 INFO:teuthology.lock.ops:Updating [smithi023.front.sepia.ceph.com]: set os type and version on server 2021-02-23T03:56:56.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-23T03:56:56.743 INFO:teuthology.orchestra.run.smithi023.stdout:x86_64 2021-02-23T03:56:56.801 INFO:teuthology.orchestra.run.smithi023.stdout:NAME="CentOS Linux" 2021-02-23T03:56:56.801 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION="7 (Core)" 2021-02-23T03:56:56.802 INFO:teuthology.orchestra.run.smithi023.stdout:ID="centos" 2021-02-23T03:56:56.802 INFO:teuthology.orchestra.run.smithi023.stdout:ID_LIKE="rhel fedora" 2021-02-23T03:56:56.802 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_ID="7" 2021-02-23T03:56:56.803 INFO:teuthology.orchestra.run.smithi023.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2021-02-23T03:56:56.803 INFO:teuthology.orchestra.run.smithi023.stdout:ANSI_COLOR="0;31" 2021-02-23T03:56:56.803 INFO:teuthology.orchestra.run.smithi023.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2021-02-23T03:56:56.803 INFO:teuthology.orchestra.run.smithi023.stdout:HOME_URL="https://www.centos.org/" 2021-02-23T03:56:56.804 INFO:teuthology.orchestra.run.smithi023.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-02-23T03:56:56.804 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-02-23T03:56:56.804 INFO:teuthology.orchestra.run.smithi023.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2021-02-23T03:56:56.804 INFO:teuthology.orchestra.run.smithi023.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2021-02-23T03:56:56.805 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-02-23T03:56:56.805 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2021-02-23T03:56:56.805 INFO:teuthology.orchestra.run.smithi023.stdout: 2021-02-23T03:56:56.806 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2021-02-23T03:56:56.823 INFO:teuthology.lock.ops:Updating [smithi041.front.sepia.ceph.com]: set os type and version on server 2021-02-23T03:56:56.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-23T03:56:57.436 INFO:teuthology.orchestra.run.smithi041.stdout:x86_64 2021-02-23T03:56:57.512 INFO:teuthology.orchestra.run.smithi041.stdout:NAME="CentOS Linux" 2021-02-23T03:56:57.512 INFO:teuthology.orchestra.run.smithi041.stdout:VERSION="7 (Core)" 2021-02-23T03:56:57.512 INFO:teuthology.orchestra.run.smithi041.stdout:ID="centos" 2021-02-23T03:56:57.513 INFO:teuthology.orchestra.run.smithi041.stdout:ID_LIKE="rhel fedora" 2021-02-23T03:56:57.513 INFO:teuthology.orchestra.run.smithi041.stdout:VERSION_ID="7" 2021-02-23T03:56:57.513 INFO:teuthology.orchestra.run.smithi041.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2021-02-23T03:56:57.513 INFO:teuthology.orchestra.run.smithi041.stdout:ANSI_COLOR="0;31" 2021-02-23T03:56:57.514 INFO:teuthology.orchestra.run.smithi041.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2021-02-23T03:56:57.514 INFO:teuthology.orchestra.run.smithi041.stdout:HOME_URL="https://www.centos.org/" 2021-02-23T03:56:57.514 INFO:teuthology.orchestra.run.smithi041.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-02-23T03:56:57.514 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-02-23T03:56:57.514 INFO:teuthology.orchestra.run.smithi041.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2021-02-23T03:56:57.515 INFO:teuthology.orchestra.run.smithi041.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2021-02-23T03:56:57.515 INFO:teuthology.orchestra.run.smithi041.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-02-23T03:56:57.515 INFO:teuthology.orchestra.run.smithi041.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2021-02-23T03:56:57.515 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-02-23T03:56:57.516 INFO:teuthology.lock.ops:Updating smithi041.front.sepia.ceph.com on lock server 2021-02-23T03:56:57.531 INFO:teuthology.lock.ops:Updating [smithi066.front.sepia.ceph.com]: set os type and version on server 2021-02-23T03:56:57.532 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-23T03:56:58.099 INFO:teuthology.orchestra.run.smithi066.stdout:x86_64 2021-02-23T03:56:58.204 INFO:teuthology.orchestra.run.smithi066.stdout:NAME="CentOS Linux" 2021-02-23T03:56:58.204 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION="7 (Core)" 2021-02-23T03:56:58.204 INFO:teuthology.orchestra.run.smithi066.stdout:ID="centos" 2021-02-23T03:56:58.204 INFO:teuthology.orchestra.run.smithi066.stdout:ID_LIKE="rhel fedora" 2021-02-23T03:56:58.205 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_ID="7" 2021-02-23T03:56:58.205 INFO:teuthology.orchestra.run.smithi066.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2021-02-23T03:56:58.205 INFO:teuthology.orchestra.run.smithi066.stdout:ANSI_COLOR="0;31" 2021-02-23T03:56:58.205 INFO:teuthology.orchestra.run.smithi066.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2021-02-23T03:56:58.205 INFO:teuthology.orchestra.run.smithi066.stdout:HOME_URL="https://www.centos.org/" 2021-02-23T03:56:58.206 INFO:teuthology.orchestra.run.smithi066.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-02-23T03:56:58.206 INFO:teuthology.orchestra.run.smithi066.stdout: 2021-02-23T03:56:58.206 INFO:teuthology.orchestra.run.smithi066.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2021-02-23T03:56:58.206 INFO:teuthology.orchestra.run.smithi066.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2021-02-23T03:56:58.206 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-02-23T03:56:58.207 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2021-02-23T03:56:58.207 INFO:teuthology.orchestra.run.smithi066.stdout: 2021-02-23T03:56:58.208 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server 2021-02-23T03:56:58.231 INFO:teuthology.lock.ops:Updating [smithi148.front.sepia.ceph.com]: set os type and version on server 2021-02-23T03:56:58.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-23T03:56:58.844 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2021-02-23T03:56:58.928 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Linux" 2021-02-23T03:56:58.928 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="7 (Core)" 2021-02-23T03:56:58.928 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos" 2021-02-23T03:56:58.929 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora" 2021-02-23T03:56:58.929 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="7" 2021-02-23T03:56:58.929 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2021-02-23T03:56:58.929 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2021-02-23T03:56:58.930 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2021-02-23T03:56:58.930 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://www.centos.org/" 2021-02-23T03:56:58.930 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-02-23T03:56:58.930 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-02-23T03:56:58.930 INFO:teuthology.orchestra.run.smithi148.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2021-02-23T03:56:58.931 INFO:teuthology.orchestra.run.smithi148.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2021-02-23T03:56:58.931 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-02-23T03:56:58.931 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2021-02-23T03:56:58.931 INFO:teuthology.orchestra.run.smithi148.stdout: 2021-02-23T03:56:58.932 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2021-02-23T03:56:58.948 INFO:teuthology.lock.ops:Updating [smithi178.front.sepia.ceph.com]: set os type and version on server 2021-02-23T03:56:58.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-23T03:56:59.528 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2021-02-23T03:56:59.597 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="CentOS Linux" 2021-02-23T03:56:59.597 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="7 (Core)" 2021-02-23T03:56:59.597 INFO:teuthology.orchestra.run.smithi178.stdout:ID="centos" 2021-02-23T03:56:59.598 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE="rhel fedora" 2021-02-23T03:56:59.598 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="7" 2021-02-23T03:56:59.598 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2021-02-23T03:56:59.598 INFO:teuthology.orchestra.run.smithi178.stdout:ANSI_COLOR="0;31" 2021-02-23T03:56:59.599 INFO:teuthology.orchestra.run.smithi178.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2021-02-23T03:56:59.599 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://www.centos.org/" 2021-02-23T03:56:59.599 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-02-23T03:56:59.599 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-02-23T03:56:59.599 INFO:teuthology.orchestra.run.smithi178.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2021-02-23T03:56:59.600 INFO:teuthology.orchestra.run.smithi178.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2021-02-23T03:56:59.600 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-02-23T03:56:59.600 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2021-02-23T03:56:59.600 INFO:teuthology.orchestra.run.smithi178.stdout: 2021-02-23T03:56:59.601 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2021-02-23T03:56:59.697 INFO:teuthology.dispatcher.supervisor:Running job 5905613 2021-02-23T03:56:59.698 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-02-23_02:30:05-upgrade:mimic-x-nautilus-distro-basic-smithi/5905613 --name teuthology-2021-02-23_02:30:05-upgrade:mimic-x-nautilus-distro-basic-smithi --description upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-nautilus 6.1-msgr2 7-final-workload/{rbd-python rgw-swift-ragweed_check snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/centos_latest thrashosds-health} -- /home/teuthworker/archive/teuthology-2021-02-23_02:30:05-upgrade:mimic-x-nautilus-distro-basic-smithi/5905613/orig.config.yaml 2021-02-23T03:56:59.701 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-02-23_02:30:05-upgrade:mimic-x-nautilus-distro-basic-smithi/5905613 2021-02-23T03:56:59.701 INFO:teuthology.dispatcher.supervisor:Job PID: 18918 2021-02-23T03:56:59.702 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-02-23T15:57:19.239 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-02-23T15:57:21.467 INFO:teuthology.kill:Killing Pids: {18918} 2021-02-23T15:57:23.488 INFO:teuthology.kill:Nuking machines: ['smithi023', 'smithi041', 'smithi066', 'smithi148', 'smithi178'] 2021-02-23T15:57:24.328 INFO:teuthology.kill:2021-02-23 15:57:24,328.328 INFO:teuthology.nuke:targets: 2021-02-23T15:57:24.330 INFO:teuthology.kill: smithi023.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCwTDId6OAWKb0GSe9fEhnp5FAL17bNEiUKEoJqjtiRlhO4f39lC52hTRYfsL8WUcptHp9YZ72Vi9bYPwwRXZ8hu9pIO6Uc52xUw8OxRQXzIRmNYiDW92I8rT0L5qN0RzCpPiKAi0vglQBoVedK36q5jTlKlMwgYEK/Pp6i+IxJlKlIkzZkTgaUbLAGjOJfaZJdvLhWBZFULxikcHNmIBnwiHbOp+jcWouVT/Ejqvc+k4VxjbCiuYY9Bu3T1JCNCOwf93FV+v2xnIZfdw13VeydL5Y4nS6SgqnXR+aFBDmJ8NPop6gxhdIUolXY5YZ2TSN+3ZXyWatkSDN2IoGvuu9b 2021-02-23T15:57:24.330 INFO:teuthology.kill: smithi041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCncs1+1EKb2/W4zSQidbaZWCFIdeebSjWQ0+9nsLTVblJPXpNCYwxaNKlARXa1YnAZklHUMbhU2zP/gh91m7C+4goe7QAGFBzlnhUjnvLlOGWRAfSqClIX+vvuZULAfB8swFZo1bps5AJFOJUVT9SkPFnC/nY2BRJG4fD+Po9E8NVaHTLJ3YsGmCsjtjL72KwGFxkFPWvwlnWFsgnjbNbC6Fd/nK62UVOQE4FIhlbt7v68w9a1todfQ4P68UvYzbeLVYA0y3AlhnbQeT7+KwE6mCbG4j8OWk0FBuqKXOm327e+js2mwQpJS1QkfoPV0PFC23vZOEk5eftSoFtKg3Nb 2021-02-23T15:57:24.330 INFO:teuthology.kill: smithi066.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkGi5bzYkNUFEY/M5crudcoXdyYDpyt01dhNIqCarDWCdxcdJmYwArMNvErAxyh+r6v0MEeMkGfZsGaF1npiDZFmv2mbhdYuA0XEVCHvnj8pkwsSo8usBXAKGIqhos7NwAuhHV+l4WB4524wex+St7/5OGEZiRllGruqAfXBn+DDqs6K917fTESmWsmKF0hz79o7J+UDkw5W6adSzWb2ehGS6R3FG1yf5DRkRFSibzdqwmH/rEcnH9TJ8b/rCXwCFBQyd2OvsEXp49X2U7S1RNrnIxrXu4Uqeb4twJF9mL+EOUJq6igG63ARtcC14mRjZQ1WKQ6NzWm02zbZDG7QEF 2021-02-23T15:57:24.331 INFO:teuthology.kill: smithi148.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxxJIHIWgJELNVxcKa5ZVAqbbqxFNizrakTrsbA4a62eCcoZOMt9tHa4U/eNUJJWHO+kXOnyl6F8BVNVWjb96ikaFrbZfYqkkapGlU+IXtcvX5+SsS15Bqd7u0ZmFs7Se7/vEYzyK4Dj9EWM28aAAO0jChpQrunMppidns6gcG4tnW7DGF1Pl2Q+MITwptri7bgL4Mo32COwhHfLBYsVe2i1Uy88JYUj/A2A9f4Bzl0VaYAs9EJceeKPlTT0vdIoiwpRxI/FyHI5O2XwkOz+jLQHR3B7WOU0NPs+39pjEPyBZP5A+FQOF1L8ixG9WYba4uIh5ITXkf37axS4NlRgnt 2021-02-23T15:57:24.332 INFO:teuthology.kill: smithi178.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0XKHieOcheTGMqeP6DWduHccL+o+BP0S9gFlpXEnOa2UeOB+/a0cIsLPsLu5Etjl9sI9BjlUIIys2pmziUc2ZDgoA8Welcj/rySZSEH+khcjiiDwhKV/ZCxKU//Zv0nvtinbDl19g3sgDZI/uTW/lSTX9ELnCcoMeEPR1WlRiJWQHHgIxmFdNM7C7hw8CnV66HhzvjkGTAidw0dg4I44XrjerhqGmjYBrOZwwFy2GzOOZeiD2hyH5THtAnWtkov2INJ82QxGqrHNaoWXbRUA40CMlR38fRX/tbPL6vk5E2G+11qVXI0x0R7QZsBbmhxHYzegfXGe3oMEe7psLEsDf 2021-02-23T15:57:24.332 INFO:teuthology.kill:2021-02-23 15:57:24,328.328 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:57:24.334 INFO:teuthology.kill:2021-02-23 15:57:24,334.334 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:57:24.336 INFO:teuthology.kill:2021-02-23 15:57:24,336.336 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:57:24.338 INFO:teuthology.kill:2021-02-23 15:57:24,338.338 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:57:24.340 INFO:teuthology.kill:2021-02-23 15:57:24,340.340 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:57:24.415 INFO:teuthology.kill:2021-02-23 15:57:24,415.415 INFO:teuthology.orchestra.console:Power off smithi066 2021-02-23T15:57:24.429 INFO:teuthology.kill:2021-02-23 15:57:24,429.429 INFO:teuthology.orchestra.console:Power off smithi023 2021-02-23T15:57:24.442 INFO:teuthology.kill:2021-02-23 15:57:24,442.442 INFO:teuthology.orchestra.console:Power off smithi178 2021-02-23T15:57:24.455 INFO:teuthology.kill:2021-02-23 15:57:24,455.455 INFO:teuthology.orchestra.console:Power off smithi148 2021-02-23T15:57:24.471 INFO:teuthology.kill:2021-02-23 15:57:24,471.471 INFO:teuthology.orchestra.console:Power off smithi041 2021-02-23T15:57:32.838 INFO:teuthology.kill:2021-02-23 15:57:32,837.837 INFO:teuthology.orchestra.console:Power off for smithi178 completed 2021-02-23T15:57:32.845 INFO:teuthology.kill:2021-02-23 15:57:32,845.845 INFO:teuthology.orchestra.console:Power off for smithi023 completed 2021-02-23T15:57:32.854 INFO:teuthology.kill:2021-02-23 15:57:32,853.853 INFO:teuthology.orchestra.console:Power off for smithi148 completed 2021-02-23T15:57:32.861 INFO:teuthology.kill:2021-02-23 15:57:32,861.861 INFO:teuthology.orchestra.console:Power off for smithi066 completed 2021-02-23T15:57:32.870 INFO:teuthology.kill:2021-02-23 15:57:32,870.870 INFO:teuthology.orchestra.console:Power off for smithi041 completed 2021-02-23T15:57:33.180 INFO:teuthology.task.internal:roles: ubuntu@smithi023.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2021-02-23T15:57:33.184 INFO:teuthology.task.internal:roles: ubuntu@smithi041.front.sepia.ceph.com - ['mon.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2021-02-23T15:57:33.184 INFO:teuthology.task.internal:roles: ubuntu@smithi066.front.sepia.ceph.com - ['mon.c'] 2021-02-23T15:57:33.185 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11'] 2021-02-23T15:57:33.187 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['client.0'] 2021-02-23T15:57:33.190 INFO:teuthology.misc:Compressing logs... 2021-02-23T15:57:33.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-23T15:58:33.333 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-02-23T15:58:36.129 INFO:teuthology.kill:No teuthology processes running 2021-02-23T15:58:37.465 INFO:teuthology.kill:Nuking machines: ['smithi023', 'smithi041', 'smithi066', 'smithi148', 'smithi178'] 2021-02-23T15:58:38.300 INFO:teuthology.kill:2021-02-23 15:58:38,300.300 INFO:teuthology.nuke:targets: 2021-02-23T15:58:38.301 INFO:teuthology.kill: smithi023.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCwTDId6OAWKb0GSe9fEhnp5FAL17bNEiUKEoJqjtiRlhO4f39lC52hTRYfsL8WUcptHp9YZ72Vi9bYPwwRXZ8hu9pIO6Uc52xUw8OxRQXzIRmNYiDW92I8rT0L5qN0RzCpPiKAi0vglQBoVedK36q5jTlKlMwgYEK/Pp6i+IxJlKlIkzZkTgaUbLAGjOJfaZJdvLhWBZFULxikcHNmIBnwiHbOp+jcWouVT/Ejqvc+k4VxjbCiuYY9Bu3T1JCNCOwf93FV+v2xnIZfdw13VeydL5Y4nS6SgqnXR+aFBDmJ8NPop6gxhdIUolXY5YZ2TSN+3ZXyWatkSDN2IoGvuu9b 2021-02-23T15:58:38.301 INFO:teuthology.kill: smithi041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCncs1+1EKb2/W4zSQidbaZWCFIdeebSjWQ0+9nsLTVblJPXpNCYwxaNKlARXa1YnAZklHUMbhU2zP/gh91m7C+4goe7QAGFBzlnhUjnvLlOGWRAfSqClIX+vvuZULAfB8swFZo1bps5AJFOJUVT9SkPFnC/nY2BRJG4fD+Po9E8NVaHTLJ3YsGmCsjtjL72KwGFxkFPWvwlnWFsgnjbNbC6Fd/nK62UVOQE4FIhlbt7v68w9a1todfQ4P68UvYzbeLVYA0y3AlhnbQeT7+KwE6mCbG4j8OWk0FBuqKXOm327e+js2mwQpJS1QkfoPV0PFC23vZOEk5eftSoFtKg3Nb 2021-02-23T15:58:38.302 INFO:teuthology.kill: smithi066.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkGi5bzYkNUFEY/M5crudcoXdyYDpyt01dhNIqCarDWCdxcdJmYwArMNvErAxyh+r6v0MEeMkGfZsGaF1npiDZFmv2mbhdYuA0XEVCHvnj8pkwsSo8usBXAKGIqhos7NwAuhHV+l4WB4524wex+St7/5OGEZiRllGruqAfXBn+DDqs6K917fTESmWsmKF0hz79o7J+UDkw5W6adSzWb2ehGS6R3FG1yf5DRkRFSibzdqwmH/rEcnH9TJ8b/rCXwCFBQyd2OvsEXp49X2U7S1RNrnIxrXu4Uqeb4twJF9mL+EOUJq6igG63ARtcC14mRjZQ1WKQ6NzWm02zbZDG7QEF 2021-02-23T15:58:38.302 INFO:teuthology.kill: smithi148.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxxJIHIWgJELNVxcKa5ZVAqbbqxFNizrakTrsbA4a62eCcoZOMt9tHa4U/eNUJJWHO+kXOnyl6F8BVNVWjb96ikaFrbZfYqkkapGlU+IXtcvX5+SsS15Bqd7u0ZmFs7Se7/vEYzyK4Dj9EWM28aAAO0jChpQrunMppidns6gcG4tnW7DGF1Pl2Q+MITwptri7bgL4Mo32COwhHfLBYsVe2i1Uy88JYUj/A2A9f4Bzl0VaYAs9EJceeKPlTT0vdIoiwpRxI/FyHI5O2XwkOz+jLQHR3B7WOU0NPs+39pjEPyBZP5A+FQOF1L8ixG9WYba4uIh5ITXkf37axS4NlRgnt 2021-02-23T15:58:38.303 INFO:teuthology.kill: smithi178.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0XKHieOcheTGMqeP6DWduHccL+o+BP0S9gFlpXEnOa2UeOB+/a0cIsLPsLu5Etjl9sI9BjlUIIys2pmziUc2ZDgoA8Welcj/rySZSEH+khcjiiDwhKV/ZCxKU//Zv0nvtinbDl19g3sgDZI/uTW/lSTX9ELnCcoMeEPR1WlRiJWQHHgIxmFdNM7C7hw8CnV66HhzvjkGTAidw0dg4I44XrjerhqGmjYBrOZwwFy2GzOOZeiD2hyH5THtAnWtkov2INJ82QxGqrHNaoWXbRUA40CMlR38fRX/tbPL6vk5E2G+11qVXI0x0R7QZsBbmhxHYzegfXGe3oMEe7psLEsDf 2021-02-23T15:58:38.329 INFO:teuthology.kill:2021-02-23 15:58:38,328.328 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:58:38.331 INFO:teuthology.kill:2021-02-23 15:58:38,331.331 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:58:38.335 INFO:teuthology.kill:2021-02-23 15:58:38,334.334 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:58:38.338 INFO:teuthology.kill:2021-02-23 15:58:38,338.338 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:58:38.344 INFO:teuthology.kill:2021-02-23 15:58:38,343.343 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T15:58:38.416 INFO:teuthology.kill:2021-02-23 15:58:38,416.416 INFO:teuthology.orchestra.console:Power off smithi023 2021-02-23T15:58:38.435 INFO:teuthology.kill:2021-02-23 15:58:38,435.435 INFO:teuthology.orchestra.console:Power off smithi148 2021-02-23T15:58:38.453 INFO:teuthology.kill:2021-02-23 15:58:38,453.453 INFO:teuthology.orchestra.console:Power off smithi178 2021-02-23T15:58:38.467 INFO:teuthology.kill:2021-02-23 15:58:38,466.466 INFO:teuthology.orchestra.console:Power off smithi041 2021-02-23T15:58:38.488 INFO:teuthology.kill:2021-02-23 15:58:38,488.488 INFO:teuthology.orchestra.console:Power off smithi066 2021-02-23T15:58:38.632 INFO:teuthology.kill:2021-02-23 15:58:38,631.631 INFO:teuthology.orchestra.console:Power off for smithi178 completed 2021-02-23T15:58:38.632 INFO:teuthology.kill:2021-02-23 15:58:38,632.632 INFO:teuthology.orchestra.console:Power off for smithi148 completed 2021-02-23T15:58:38.633 INFO:teuthology.kill:2021-02-23 15:58:38,633.633 INFO:teuthology.orchestra.console:Power off for smithi041 completed 2021-02-23T15:58:38.639 INFO:teuthology.kill:2021-02-23 15:58:38,638.638 INFO:teuthology.orchestra.console:Power off for smithi023 completed 2021-02-23T15:58:38.645 INFO:teuthology.kill:2021-02-23 15:58:38,645.645 INFO:teuthology.orchestra.console:Power off for smithi066 completed 2021-02-23T15:58:38.784 INFO:teuthology.kill:2021-02-23 15:58:38,784.784 INFO:teuthology.lock.ops:unlocked smithi178.front.sepia.ceph.com 2021-02-23T15:58:38.787 INFO:teuthology.kill:2021-02-23 15:58:38,787.787 INFO:teuthology.lock.ops:unlocked smithi148.front.sepia.ceph.com 2021-02-23T15:58:38.789 INFO:teuthology.kill:2021-02-23 15:58:38,789.789 INFO:teuthology.lock.ops:unlocked smithi023.front.sepia.ceph.com 2021-02-23T15:58:38.792 INFO:teuthology.kill:2021-02-23 15:58:38,792.792 INFO:teuthology.lock.ops:unlocked smithi066.front.sepia.ceph.com 2021-02-23T15:58:38.796 INFO:teuthology.kill:2021-02-23 15:58:38,796.796 INFO:teuthology.lock.ops:unlocked smithi041.front.sepia.ceph.com 2021-02-23T16:00:38.979 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-02-23T16:00:39.172 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-02-23T16:00:39.172 INFO:teuthology.nuke:Checking targets against current locks 2021-02-23T16:00:39.460 INFO:teuthology.nuke:Not nuking smithi148.front.sepia.ceph.com because description doesn't match 2021-02-23T16:00:39.461 INFO:teuthology.nuke:Not nuking smithi178.front.sepia.ceph.com because description doesn't match 2021-02-23T16:00:39.461 INFO:teuthology.nuke:Not nuking smithi023.front.sepia.ceph.com because description doesn't match 2021-02-23T16:00:39.462 INFO:teuthology.nuke:Not nuking smithi066.front.sepia.ceph.com because description doesn't match 2021-02-23T16:00:39.462 INFO:teuthology.nuke:Not nuking smithi041.front.sepia.ceph.com because description doesn't match