2021-03-31T07:56:41.949 INFO:root:teuthology version: 1.1.0-9870012c 2021-03-31T07:56:42.056 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba014.front.sepia.ceph.com' reimaging 2021-03-31T07:56:42.058 INFO:teuthology.lock.ops:Updating [ubuntu@gibba014.front.sepia.ceph.com]: reset os type and version on server 2021-03-31T07:56:42.058 INFO:teuthology.lock.ops:Updating gibba014.front.sepia.ceph.com on lock server 2021-03-31T07:56:42.075 INFO:teuthology.lock.ops:Node 'ubuntu@gibba014.front.sepia.ceph.com' reimaging is complete 2021-03-31T07:56:42.076 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba030.front.sepia.ceph.com' reimaging 2021-03-31T07:56:42.076 INFO:teuthology.lock.ops:Updating [ubuntu@gibba030.front.sepia.ceph.com]: reset os type and version on server 2021-03-31T07:56:42.077 INFO:teuthology.lock.ops:Updating gibba030.front.sepia.ceph.com on lock server 2021-03-31T07:56:42.094 INFO:teuthology.lock.ops:Node 'ubuntu@gibba030.front.sepia.ceph.com' reimaging is complete 2021-03-31T07:56:42.313 INFO:teuthology.provision.fog.gibba030:Scheduling deploy of centos 7.6 2021-03-31T07:56:42.344 INFO:teuthology.provision.fog.gibba014:Scheduling deploy of centos 7.6 2021-03-31T07:56:42.731 INFO:teuthology.orchestra.console:Power off gibba030 2021-03-31T07:56:42.791 INFO:teuthology.orchestra.console:Power off gibba014 2021-03-31T07:56:51.159 INFO:teuthology.orchestra.console:Power off for gibba014 completed 2021-03-31T07:56:51.260 INFO:teuthology.orchestra.console:Power on gibba014 2021-03-31T07:56:55.238 INFO:teuthology.orchestra.console:Power off for gibba030 completed 2021-03-31T07:56:55.339 INFO:teuthology.orchestra.console:Power on gibba030 2021-03-31T07:56:59.629 INFO:teuthology.orchestra.console:Power on for gibba014 completed 2021-03-31T07:56:59.730 INFO:teuthology.provision.fog.gibba014:Waiting for deploy to finish 2021-03-31T07:57:03.709 INFO:teuthology.orchestra.console:Power on for gibba030 completed 2021-03-31T07:57:03.810 INFO:teuthology.provision.fog.gibba030:Waiting for deploy to finish 2021-03-31T08:01:03.011 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-31T08:01:13.920 INFO:teuthology.provision.fog.gibba014:Node is ready 2021-03-31T08:01:14.311 INFO:teuthology.orchestra.run.gibba014.stdout:gibba014.front.sepia.ceph.com 2021-03-31T08:01:14.681 INFO:teuthology.orchestra.run.gibba014.stdout:172.21.2.114 gibba014.front.sepia.ceph.com gibba014 2021-03-31T08:01:16.571 INFO:teuthology.provision.fog.gibba014:Deploy complete! 2021-03-31T08:01:29.426 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-31T08:01:30.081 INFO:teuthology.provision.fog.gibba030:Node is ready 2021-03-31T08:01:30.460 INFO:teuthology.orchestra.run.gibba030.stdout:gibba030.front.sepia.ceph.com 2021-03-31T08:01:30.829 INFO:teuthology.orchestra.run.gibba030.stdout:172.21.2.130 gibba030.front.sepia.ceph.com gibba030 2021-03-31T08:01:32.723 INFO:teuthology.provision.fog.gibba030:Deploy complete! 2021-03-31T08:01:32.883 INFO:teuthology.lock.ops:Checking gibba014.front.sepia.ceph.com 2021-03-31T08:01:32.889 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-31T08:01:32.907 INFO:teuthology.lock.ops:Checking gibba030.front.sepia.ceph.com 2021-03-31T08:01:32.907 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-31T08:01:32.923 INFO:teuthology.lock.ops:Updating [gibba014.front.sepia.ceph.com]: set os type and version on server 2021-03-31T08:01:32.923 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-31T08:01:33.940 INFO:teuthology.orchestra.run.gibba014.stdout:x86_64 2021-03-31T08:01:34.312 INFO:teuthology.orchestra.run.gibba014.stdout:NAME="CentOS Linux" 2021-03-31T08:01:34.312 INFO:teuthology.orchestra.run.gibba014.stdout:VERSION="7 (Core)" 2021-03-31T08:01:34.313 INFO:teuthology.orchestra.run.gibba014.stdout:ID="centos" 2021-03-31T08:01:34.313 INFO:teuthology.orchestra.run.gibba014.stdout:ID_LIKE="rhel fedora" 2021-03-31T08:01:34.313 INFO:teuthology.orchestra.run.gibba014.stdout:VERSION_ID="7" 2021-03-31T08:01:34.313 INFO:teuthology.orchestra.run.gibba014.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2021-03-31T08:01:34.313 INFO:teuthology.orchestra.run.gibba014.stdout:ANSI_COLOR="0;31" 2021-03-31T08:01:34.314 INFO:teuthology.orchestra.run.gibba014.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2021-03-31T08:01:34.314 INFO:teuthology.orchestra.run.gibba014.stdout:HOME_URL="https://www.centos.org/" 2021-03-31T08:01:34.314 INFO:teuthology.orchestra.run.gibba014.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-03-31T08:01:34.314 INFO:teuthology.orchestra.run.gibba014.stdout: 2021-03-31T08:01:34.315 INFO:teuthology.orchestra.run.gibba014.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2021-03-31T08:01:34.315 INFO:teuthology.orchestra.run.gibba014.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2021-03-31T08:01:34.315 INFO:teuthology.orchestra.run.gibba014.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-03-31T08:01:34.315 INFO:teuthology.orchestra.run.gibba014.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2021-03-31T08:01:34.315 INFO:teuthology.orchestra.run.gibba014.stdout: 2021-03-31T08:01:34.317 INFO:teuthology.lock.ops:Updating gibba014.front.sepia.ceph.com on lock server 2021-03-31T08:01:34.332 INFO:teuthology.lock.ops:Updating [gibba030.front.sepia.ceph.com]: set os type and version on server 2021-03-31T08:01:34.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-31T08:01:35.346 INFO:teuthology.orchestra.run.gibba030.stdout:x86_64 2021-03-31T08:01:35.717 INFO:teuthology.orchestra.run.gibba030.stdout:NAME="CentOS Linux" 2021-03-31T08:01:35.717 INFO:teuthology.orchestra.run.gibba030.stdout:VERSION="7 (Core)" 2021-03-31T08:01:35.717 INFO:teuthology.orchestra.run.gibba030.stdout:ID="centos" 2021-03-31T08:01:35.718 INFO:teuthology.orchestra.run.gibba030.stdout:ID_LIKE="rhel fedora" 2021-03-31T08:01:35.718 INFO:teuthology.orchestra.run.gibba030.stdout:VERSION_ID="7" 2021-03-31T08:01:35.718 INFO:teuthology.orchestra.run.gibba030.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2021-03-31T08:01:35.718 INFO:teuthology.orchestra.run.gibba030.stdout:ANSI_COLOR="0;31" 2021-03-31T08:01:35.718 INFO:teuthology.orchestra.run.gibba030.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2021-03-31T08:01:35.719 INFO:teuthology.orchestra.run.gibba030.stdout:HOME_URL="https://www.centos.org/" 2021-03-31T08:01:35.719 INFO:teuthology.orchestra.run.gibba030.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-03-31T08:01:35.719 INFO:teuthology.orchestra.run.gibba030.stdout: 2021-03-31T08:01:35.719 INFO:teuthology.orchestra.run.gibba030.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2021-03-31T08:01:35.720 INFO:teuthology.orchestra.run.gibba030.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2021-03-31T08:01:35.720 INFO:teuthology.orchestra.run.gibba030.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-03-31T08:01:35.720 INFO:teuthology.orchestra.run.gibba030.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2021-03-31T08:01:35.720 INFO:teuthology.orchestra.run.gibba030.stdout: 2021-03-31T08:01:35.721 INFO:teuthology.lock.ops:Updating gibba030.front.sepia.ceph.com on lock server 2021-03-31T08:01:35.796 INFO:teuthology.dispatcher.supervisor:Running job 6012505 2021-03-31T08:01:35.796 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_9870012c9b43ad9e4e27984868ce0706e230f04f/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-03-31_03:30:03-rados-octopus-distro-basic-gibba/6012505 --name teuthology-2021-03-31_03:30:03-rados-octopus-distro-basic-gibba --description rados/cephadm/upgrade/{1-start 2-repo_digest/defaut 3-start-upgrade 4-wait distro$/{centos_7} fixed-2} -- /home/teuthworker/archive/teuthology-2021-03-31_03:30:03-rados-octopus-distro-basic-gibba/6012505/orig.config.yaml 2021-03-31T08:01:35.800 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-03-31_03:30:03-rados-octopus-distro-basic-gibba/6012505 2021-03-31T08:01:35.801 INFO:teuthology.dispatcher.supervisor:Job PID: 3824 2021-03-31T08:01:35.801 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-03-31T20:02:17.850 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-31T20:02:18.065 INFO:teuthology.kill:Killing Pids: {3824} 2021-03-31T20:02:19.899 INFO:teuthology.kill:Nuking machines: ['gibba014', 'gibba030'] 2021-03-31T20:02:20.878 INFO:teuthology.kill:2021-03-31 20:02:20,878.878 INFO:teuthology.nuke:targets: 2021-03-31T20:02:20.879 INFO:teuthology.kill: gibba014.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkAFVZbOuzqoqEy9Q7+XLwuOOuPM2hucEerFZBFnfe4lRLJnjtXBItrdAoJLZwTLVyveS21DIYU1BPJxbDkkRUGhMN5ccuu0oaTcYkdXp0lZieRQM4nU6CBi+Dz+Q4R2GPvlMNwbFJwGw7AHfsNpJrBI/z8hGBfhMX2lsiyVlR+gPRFD9UVmLeXP/KwbOUGQBsPDfl0ETSR/UoAEf+Yt7wqCSX8VA4vSoKLmF1NPv2Selhuadt/wSxHFSuSKEiik23TqPaNt5DrnmoylFg9JBLaKHrfkEMm11aYF/8jKLK7Yh4Gb2b5vCLB30yQUE0cu/yZg9o24IAq1VClKPCYtCV 2021-03-31T20:02:20.880 INFO:teuthology.kill: gibba030.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCorFeMC5+t79rRCXSs2AYN/il7ZKHO5B3a/omp6YYSqgQGm7lsw/qJZW1Cv2GDMx25WyzihoGtKURCRSxVFdO+0DaIDG+ST4Q69FJTifmYgr7RU29YkZyIfOQoDGygVg1wjiMAKdlcppN+9Jmh6uLZal2rcvJloIw00aOuVDPRv3rhhryGIIHX59TW+8Z+PPPIhdLckjQXc6SD7qn9NDuiHbnOXBGRBRoeChAJOnbS81slj1ZOdw/N96I617gFKv4UBZ+DkDqttpWYOu5/KNzbQbBh5Ju/8TRZ3EqyeiZm1Ioy4W4YHJ8x5/skHh4e2VX+PXDMRySPd9SSqds9ZkYT 2021-03-31T20:02:20.880 INFO:teuthology.kill:2021-03-31 20:02:20,878.878 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-31T20:02:20.881 INFO:teuthology.kill:2021-03-31 20:02:20,880.880 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-31T20:02:20.997 INFO:teuthology.kill:2021-03-31 20:02:20,996.996 INFO:teuthology.orchestra.console:Power off gibba014 2021-03-31T20:02:21.016 INFO:teuthology.kill:2021-03-31 20:02:21,016.016 INFO:teuthology.orchestra.console:Power off gibba030 2021-03-31T20:02:33.532 INFO:teuthology.kill:2021-03-31 20:02:33,532.532 INFO:teuthology.orchestra.console:Power off for gibba014 completed 2021-03-31T20:02:33.612 INFO:teuthology.kill:2021-03-31 20:02:33,611.611 INFO:teuthology.orchestra.console:Power off for gibba030 completed 2021-03-31T20:02:33.829 INFO:teuthology.task.internal:roles: ubuntu@gibba014.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2021-03-31T20:02:33.830 INFO:teuthology.task.internal:roles: ubuntu@gibba030.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2021-03-31T20:02:33.831 INFO:teuthology.misc:Compressing logs... 2021-03-31T20:02:33.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-31T20:03:33.891 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_9870012c9b43ad9e4e27984868ce0706e230f04f/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9870012c9b43ad9e4e27984868ce0706e230f04f/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9870012c9b43ad9e4e27984868ce0706e230f04f/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_9870012c9b43ad9e4e27984868ce0706e230f04f/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_9870012c9b43ad9e4e27984868ce0706e230f04f/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_9870012c9b43ad9e4e27984868ce0706e230f04f/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9870012c9b43ad9e4e27984868ce0706e230f04f/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-31T20:03:33.939 INFO:teuthology.kill:No teuthology processes running 2021-03-31T20:03:35.795 INFO:teuthology.kill:Nuking machines: ['gibba014', 'gibba030'] 2021-03-31T20:03:36.877 INFO:teuthology.kill:2021-03-31 20:03:36,877.877 INFO:teuthology.nuke:targets: 2021-03-31T20:03:36.878 INFO:teuthology.kill: gibba014.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkAFVZbOuzqoqEy9Q7+XLwuOOuPM2hucEerFZBFnfe4lRLJnjtXBItrdAoJLZwTLVyveS21DIYU1BPJxbDkkRUGhMN5ccuu0oaTcYkdXp0lZieRQM4nU6CBi+Dz+Q4R2GPvlMNwbFJwGw7AHfsNpJrBI/z8hGBfhMX2lsiyVlR+gPRFD9UVmLeXP/KwbOUGQBsPDfl0ETSR/UoAEf+Yt7wqCSX8VA4vSoKLmF1NPv2Selhuadt/wSxHFSuSKEiik23TqPaNt5DrnmoylFg9JBLaKHrfkEMm11aYF/8jKLK7Yh4Gb2b5vCLB30yQUE0cu/yZg9o24IAq1VClKPCYtCV 2021-03-31T20:03:36.879 INFO:teuthology.kill: gibba030.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCorFeMC5+t79rRCXSs2AYN/il7ZKHO5B3a/omp6YYSqgQGm7lsw/qJZW1Cv2GDMx25WyzihoGtKURCRSxVFdO+0DaIDG+ST4Q69FJTifmYgr7RU29YkZyIfOQoDGygVg1wjiMAKdlcppN+9Jmh6uLZal2rcvJloIw00aOuVDPRv3rhhryGIIHX59TW+8Z+PPPIhdLckjQXc6SD7qn9NDuiHbnOXBGRBRoeChAJOnbS81slj1ZOdw/N96I617gFKv4UBZ+DkDqttpWYOu5/KNzbQbBh5Ju/8TRZ3EqyeiZm1Ioy4W4YHJ8x5/skHh4e2VX+PXDMRySPd9SSqds9ZkYT 2021-03-31T20:03:38.103 INFO:teuthology.kill:2021-03-31 20:03:38,103.103 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-31T20:03:38.115 INFO:teuthology.kill:2021-03-31 20:03:38,115.115 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-31T20:03:38.516 INFO:teuthology.kill:2021-03-31 20:03:38,516.516 INFO:teuthology.orchestra.console:Power off gibba014 2021-03-31T20:03:38.541 INFO:teuthology.kill:2021-03-31 20:03:38,539.539 INFO:teuthology.orchestra.console:Power off gibba030 2021-03-31T20:03:38.700 INFO:teuthology.kill:2021-03-31 20:03:38,700.700 INFO:teuthology.orchestra.console:Power off for gibba030 completed 2021-03-31T20:03:38.707 INFO:teuthology.kill:2021-03-31 20:03:38,707.707 INFO:teuthology.orchestra.console:Power off for gibba014 completed 2021-03-31T20:03:38.855 INFO:teuthology.kill:2021-03-31 20:03:38,855.855 INFO:teuthology.lock.ops:unlocked gibba030.front.sepia.ceph.com 2021-03-31T20:03:38.873 INFO:teuthology.kill:2021-03-31 20:03:38,873.873 INFO:teuthology.lock.ops:unlocked gibba014.front.sepia.ceph.com 2021-03-31T20:05:39.237 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-03-31T20:05:39.398 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-03-31T20:05:39.398 INFO:teuthology.nuke:Checking targets against current locks 2021-03-31T20:05:39.544 INFO:teuthology.nuke:Not nuking gibba014.front.sepia.ceph.com because description doesn't match 2021-03-31T20:05:39.544 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_9870012c9b43ad9e4e27984868ce0706e230f04f/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9870012c9b43ad9e4e27984868ce0706e230f04f/scripts/dispatcher.py", line 33, in main teuthology.dispatcher.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9870012c9b43ad9e4e27984868ce0706e230f04f/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9870012c9b43ad9e4e27984868ce0706e230f04f/teuthology/dispatcher/supervisor.py", line 59, in main verbose File "/home/teuthworker/src/git.ceph.com_git_teuthology_9870012c9b43ad9e4e27984868ce0706e230f04f/teuthology/dispatcher/supervisor.py", line 152, in run_job unlock_targets(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9870012c9b43ad9e4e27984868ce0706e230f04f/teuthology/dispatcher/supervisor.py", line 196, in unlock_targets nuke(fake_ctx, True) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9870012c9b43ad9e4e27984868ce0706e230f04f/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable