2021-09-09T02:43:13.350 INFO:root:teuthology version: 1.1.0-df69d6db 2021-09-09T02:43:13.469 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba005.front.sepia.ceph.com' reimaging 2021-09-09T02:43:13.470 INFO:teuthology.lock.ops:Updating [ubuntu@gibba005.front.sepia.ceph.com]: reset os type and version on server 2021-09-09T02:43:13.470 INFO:teuthology.lock.ops:Updating gibba005.front.sepia.ceph.com on lock server 2021-09-09T02:43:13.486 INFO:teuthology.lock.ops:Node 'ubuntu@gibba005.front.sepia.ceph.com' reimaging is complete 2021-09-09T02:43:13.486 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba041.front.sepia.ceph.com' reimaging 2021-09-09T02:43:13.487 INFO:teuthology.lock.ops:Updating [ubuntu@gibba041.front.sepia.ceph.com]: reset os type and version on server 2021-09-09T02:43:13.487 INFO:teuthology.lock.ops:Updating gibba041.front.sepia.ceph.com on lock server 2021-09-09T02:43:13.502 INFO:teuthology.lock.ops:Node 'ubuntu@gibba041.front.sepia.ceph.com' reimaging is complete 2021-09-09T02:43:13.691 INFO:teuthology.provision.fog.gibba041:Scheduling deploy of centos 8.2 2021-09-09T02:43:13.700 INFO:teuthology.provision.fog.gibba005:Scheduling deploy of centos 8.2 2021-09-09T02:43:14.028 INFO:teuthology.orchestra.console:Power off gibba041 2021-09-09T02:43:14.070 INFO:teuthology.orchestra.console:Power off gibba005 2021-09-09T02:43:14.180 INFO:teuthology.orchestra.console:Power off for gibba041 completed 2021-09-09T02:43:14.221 INFO:teuthology.orchestra.console:Power off for gibba005 completed 2021-09-09T02:43:14.282 INFO:teuthology.orchestra.console:Power on gibba041 2021-09-09T02:43:14.323 INFO:teuthology.orchestra.console:Power on gibba005 2021-09-09T02:43:22.669 INFO:teuthology.orchestra.console:Power on for gibba041 completed 2021-09-09T02:43:22.764 INFO:teuthology.orchestra.console:Power on for gibba005 completed 2021-09-09T02:43:22.770 INFO:teuthology.provision.fog.gibba041:Waiting for deploy to finish 2021-09-09T02:43:22.866 INFO:teuthology.provision.fog.gibba005:Waiting for deploy to finish 2021-09-09T02:47:25.473 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-09T02:47:35.977 INFO:teuthology.provision.fog.gibba005:Node is ready 2021-09-09T02:47:35.990 INFO:teuthology.orchestra.run.gibba005.stdout:gibba031 2021-09-09T02:47:36.043 INFO:teuthology.orchestra.run.gibba005.stdout:172.21.2.131 gibba031.front.sepia.ceph.com gibba031 2021-09-09T02:47:36.342 INFO:teuthology.provision.fog.gibba005:Deploy complete! 2021-09-09T02:47:48.819 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-09T02:47:49.068 INFO:teuthology.provision.fog.gibba041:Node is ready 2021-09-09T02:47:49.079 INFO:teuthology.orchestra.run.gibba041.stdout:gibba041.front.sepia.ceph.com 2021-09-09T02:47:49.133 INFO:teuthology.orchestra.run.gibba041.stdout:172.21.2.141 gibba041.front.sepia.ceph.com gibba041 2021-09-09T02:47:49.454 INFO:teuthology.provision.fog.gibba041:Deploy complete! 2021-09-09T02:47:49.589 INFO:teuthology.lock.ops:Checking gibba005.front.sepia.ceph.com 2021-09-09T02:47:49.590 INFO:teuthology.lock.ops:New key found. Updating... 2021-09-09T02:47:49.617 INFO:teuthology.lock.ops:Checking gibba041.front.sepia.ceph.com 2021-09-09T02:47:49.617 INFO:teuthology.lock.ops:New key found. Updating... 2021-09-09T02:47:49.639 INFO:teuthology.lock.ops:Updating [gibba005.front.sepia.ceph.com]: set os type and version on server 2021-09-09T02:47:49.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-09T02:47:49.830 INFO:teuthology.orchestra.run.gibba005.stdout:x86_64 2021-09-09T02:47:49.882 INFO:teuthology.orchestra.run.gibba005.stdout:NAME="CentOS Linux" 2021-09-09T02:47:49.882 INFO:teuthology.orchestra.run.gibba005.stdout:VERSION="8 (Core)" 2021-09-09T02:47:49.883 INFO:teuthology.orchestra.run.gibba005.stdout:ID="centos" 2021-09-09T02:47:49.883 INFO:teuthology.orchestra.run.gibba005.stdout:ID_LIKE="rhel fedora" 2021-09-09T02:47:49.884 INFO:teuthology.orchestra.run.gibba005.stdout:VERSION_ID="8" 2021-09-09T02:47:49.884 INFO:teuthology.orchestra.run.gibba005.stdout:PLATFORM_ID="platform:el8" 2021-09-09T02:47:49.885 INFO:teuthology.orchestra.run.gibba005.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-09-09T02:47:49.885 INFO:teuthology.orchestra.run.gibba005.stdout:ANSI_COLOR="0;31" 2021-09-09T02:47:49.885 INFO:teuthology.orchestra.run.gibba005.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-09-09T02:47:49.886 INFO:teuthology.orchestra.run.gibba005.stdout:HOME_URL="https://www.centos.org/" 2021-09-09T02:47:49.886 INFO:teuthology.orchestra.run.gibba005.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-09-09T02:47:49.887 INFO:teuthology.orchestra.run.gibba005.stdout: 2021-09-09T02:47:49.887 INFO:teuthology.orchestra.run.gibba005.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-09-09T02:47:49.888 INFO:teuthology.orchestra.run.gibba005.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-09-09T02:47:49.888 INFO:teuthology.orchestra.run.gibba005.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-09-09T02:47:49.888 INFO:teuthology.orchestra.run.gibba005.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-09-09T02:47:49.888 INFO:teuthology.orchestra.run.gibba005.stdout: 2021-09-09T02:47:49.890 INFO:teuthology.lock.ops:Updating gibba005.front.sepia.ceph.com on lock server 2021-09-09T02:47:49.905 INFO:teuthology.lock.ops:Updating [gibba041.front.sepia.ceph.com]: set os type and version on server 2021-09-09T02:47:49.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-09T02:47:50.134 INFO:teuthology.orchestra.run.gibba041.stdout:x86_64 2021-09-09T02:47:50.186 INFO:teuthology.orchestra.run.gibba041.stdout:NAME="CentOS Linux" 2021-09-09T02:47:50.187 INFO:teuthology.orchestra.run.gibba041.stdout:VERSION="8 (Core)" 2021-09-09T02:47:50.187 INFO:teuthology.orchestra.run.gibba041.stdout:ID="centos" 2021-09-09T02:47:50.187 INFO:teuthology.orchestra.run.gibba041.stdout:ID_LIKE="rhel fedora" 2021-09-09T02:47:50.187 INFO:teuthology.orchestra.run.gibba041.stdout:VERSION_ID="8" 2021-09-09T02:47:50.188 INFO:teuthology.orchestra.run.gibba041.stdout:PLATFORM_ID="platform:el8" 2021-09-09T02:47:50.188 INFO:teuthology.orchestra.run.gibba041.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-09-09T02:47:50.188 INFO:teuthology.orchestra.run.gibba041.stdout:ANSI_COLOR="0;31" 2021-09-09T02:47:50.188 INFO:teuthology.orchestra.run.gibba041.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-09-09T02:47:50.188 INFO:teuthology.orchestra.run.gibba041.stdout:HOME_URL="https://www.centos.org/" 2021-09-09T02:47:50.189 INFO:teuthology.orchestra.run.gibba041.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-09-09T02:47:50.189 INFO:teuthology.orchestra.run.gibba041.stdout: 2021-09-09T02:47:50.189 INFO:teuthology.orchestra.run.gibba041.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-09-09T02:47:50.189 INFO:teuthology.orchestra.run.gibba041.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-09-09T02:47:50.190 INFO:teuthology.orchestra.run.gibba041.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-09-09T02:47:50.190 INFO:teuthology.orchestra.run.gibba041.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-09-09T02:47:50.190 INFO:teuthology.orchestra.run.gibba041.stdout: 2021-09-09T02:47:50.191 INFO:teuthology.lock.ops:Updating gibba041.front.sepia.ceph.com on lock server 2021-09-09T02:47:50.273 INFO:teuthology.dispatcher.supervisor:Running job 6369153 2021-09-09T02:47:50.273 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-09-01_03:30:03-rados-octopus-distro-basic-gibba/6369153 --name teuthology-2021-09-01_03:30:03-rados-octopus-distro-basic-gibba --description rados/cephadm/upgrade/{1-start-distro/1-start-centos_8 2-repo_digest/defaut 3-start-upgrade 4-wait fixed-2} -- /home/teuthworker/archive/teuthology-2021-09-01_03:30:03-rados-octopus-distro-basic-gibba/6369153/orig.config.yaml 2021-09-09T02:47:50.276 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-09-01_03:30:03-rados-octopus-distro-basic-gibba/6369153 2021-09-09T02:47:50.277 INFO:teuthology.dispatcher.supervisor:Job PID: 15759 2021-09-09T02:47:50.278 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-09-09T14:48:35.741 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-09-09T14:48:35.918 INFO:teuthology.kill:Killing Pids: {15759} 2021-09-09T14:48:37.054 INFO:teuthology.kill:Nuking machines: ['gibba005', 'gibba041'] 2021-09-09T14:48:37.999 INFO:teuthology.kill:2021-09-09 14:48:37,998.998 INFO:teuthology.nuke:targets: 2021-09-09T14:48:38.000 INFO:teuthology.kill: gibba005.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDoP3Rvk0yYYvxOXxqvEJIzCfBvlP+ZnDE5WnVw3y2MgTWi/c41cG8nlpEL0h5Pj0dMOz/ZbyuK5lp0bQJ30r0UHlrG7r24jwe4vPIdKqkanfTm/HtNh3gt8ptNlNuqDCuo+4B1oGQ2/9Zq0/kPXE1wpE0nAqxPJARr0/PA8Ko4S47t6RYtYnuXEAtP8j3yld3O3IXJJceQCgbZFVyeZayQDVYo/rYkH+32w+f+VOeZgZq1vwbCrxkTewHDkAxJwVV5W3zPjkvDjpC7Cl88kyrbpug5VYRDsbPYfIHtGJvXlxHgpS9zV37Oqjgqw6Bw3y2NTAdc60FZF1UGLIfSVMQeVo77qdX5dxapK9qHpOpMpsKbzYeiu8yj828/hsJCnhIDZK6n+FzkxX3tg1T0rj1dU3IO16VPgLwSKQxanB0cJGQtj/RvpgRf79aOmgWnkP8SYrUImDAG9BCw1gcmdM+H2cDLJq9EwN0kTVze+v804XpPV/z2/lzTYTknxGaOVUU= 2021-09-09T14:48:38.000 INFO:teuthology.kill: gibba041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAK0CpaEGylghAtn4Wy8qsDFEAtqlerzAxlXEdhcfwBv0HDkfCbBTwknPrIu7xVXETr6WzPtts7iA/Vr9miDiJIFcKiK5zUgnBxz3hPbp0UH9IQSAA7mPTEdzCC6yJpr//Y5c2QxZmLL7qaK2BYlhcgjWnVTicBQiW5TjMmRU7oDPqP1CEC0Z8GDFqFXZYPhS7wL6xvgrV+jeleMFWYFVmcyH/VgnJH4/aJoNZ4OrC9t5EQ5gov1xVwiSzY3oySuuFZAzrhFR3bG4qU3sAapTvlEdFaGAbW9c7mZwsggLKT+/J3lherkln7obeIa0QjHbXOjq4lv4mUvUSrz2C+nXD0KWYck66PVmWUQ/JKLDXT6xfE5nlSEHir9r410pMn6mZ0RFlqrrRHh2+qJs+IrCLpIXOqrKmFwee0BecV7wc+rfLgoJKUujAf3bK47ucc8Oge9mceZuGiLWhTqc45UetepILwEKqLPdhBrUJ3V2fP8chAsnc0uOjcrVRsNeV0VM= 2021-09-09T14:48:38.000 INFO:teuthology.kill:2021-09-09 14:48:37,999.999 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-09T14:48:38.002 INFO:teuthology.kill:2021-09-09 14:48:38,002.002 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-09T14:48:38.061 INFO:teuthology.kill:2021-09-09 14:48:38,060.060 INFO:teuthology.orchestra.console:Power off gibba005 2021-09-09T14:48:38.076 INFO:teuthology.kill:2021-09-09 14:48:38,075.075 INFO:teuthology.orchestra.console:Power off gibba041 2021-09-09T14:48:46.467 INFO:teuthology.kill:2021-09-09 14:48:46,466.466 INFO:teuthology.orchestra.console:Power off for gibba041 completed 2021-09-09T14:48:50.584 INFO:teuthology.kill:2021-09-09 14:48:50,583.583 INFO:teuthology.orchestra.console:Power off for gibba005 completed 2021-09-09T14:48:50.790 INFO:teuthology.task.internal:roles: ubuntu@gibba005.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2021-09-09T14:48:50.791 INFO:teuthology.task.internal:roles: ubuntu@gibba041.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2021-09-09T14:48:50.791 INFO:teuthology.misc:Compressing logs... 2021-09-09T14:48:50.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-09T14:49:25.451 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-09-09T14:49:25.494 INFO:teuthology.kill:No teuthology processes running 2021-09-09T14:49:26.481 INFO:teuthology.kill:Nuking machines: ['gibba005', 'gibba041'] 2021-09-09T14:49:27.398 INFO:teuthology.kill:2021-09-09 14:49:27,397.397 INFO:teuthology.nuke:targets: 2021-09-09T14:49:27.398 INFO:teuthology.kill: gibba005.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDoP3Rvk0yYYvxOXxqvEJIzCfBvlP+ZnDE5WnVw3y2MgTWi/c41cG8nlpEL0h5Pj0dMOz/ZbyuK5lp0bQJ30r0UHlrG7r24jwe4vPIdKqkanfTm/HtNh3gt8ptNlNuqDCuo+4B1oGQ2/9Zq0/kPXE1wpE0nAqxPJARr0/PA8Ko4S47t6RYtYnuXEAtP8j3yld3O3IXJJceQCgbZFVyeZayQDVYo/rYkH+32w+f+VOeZgZq1vwbCrxkTewHDkAxJwVV5W3zPjkvDjpC7Cl88kyrbpug5VYRDsbPYfIHtGJvXlxHgpS9zV37Oqjgqw6Bw3y2NTAdc60FZF1UGLIfSVMQeVo77qdX5dxapK9qHpOpMpsKbzYeiu8yj828/hsJCnhIDZK6n+FzkxX3tg1T0rj1dU3IO16VPgLwSKQxanB0cJGQtj/RvpgRf79aOmgWnkP8SYrUImDAG9BCw1gcmdM+H2cDLJq9EwN0kTVze+v804XpPV/z2/lzTYTknxGaOVUU= 2021-09-09T14:49:27.399 INFO:teuthology.kill: gibba041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAK0CpaEGylghAtn4Wy8qsDFEAtqlerzAxlXEdhcfwBv0HDkfCbBTwknPrIu7xVXETr6WzPtts7iA/Vr9miDiJIFcKiK5zUgnBxz3hPbp0UH9IQSAA7mPTEdzCC6yJpr//Y5c2QxZmLL7qaK2BYlhcgjWnVTicBQiW5TjMmRU7oDPqP1CEC0Z8GDFqFXZYPhS7wL6xvgrV+jeleMFWYFVmcyH/VgnJH4/aJoNZ4OrC9t5EQ5gov1xVwiSzY3oySuuFZAzrhFR3bG4qU3sAapTvlEdFaGAbW9c7mZwsggLKT+/J3lherkln7obeIa0QjHbXOjq4lv4mUvUSrz2C+nXD0KWYck66PVmWUQ/JKLDXT6xfE5nlSEHir9r410pMn6mZ0RFlqrrRHh2+qJs+IrCLpIXOqrKmFwee0BecV7wc+rfLgoJKUujAf3bK47ucc8Oge9mceZuGiLWhTqc45UetepILwEKqLPdhBrUJ3V2fP8chAsnc0uOjcrVRsNeV0VM= 2021-09-09T14:49:27.419 INFO:teuthology.kill:2021-09-09 14:49:27,418.418 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-09T14:49:27.422 INFO:teuthology.kill:2021-09-09 14:49:27,422.422 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-09T14:49:27.480 INFO:teuthology.kill:2021-09-09 14:49:27,479.479 INFO:teuthology.orchestra.console:Power off gibba005 2021-09-09T14:49:27.493 INFO:teuthology.kill:2021-09-09 14:49:27,493.493 INFO:teuthology.orchestra.console:Power off gibba041 2021-09-09T14:49:27.633 INFO:teuthology.kill:2021-09-09 14:49:27,633.633 INFO:teuthology.orchestra.console:Power off for gibba005 completed 2021-09-09T14:49:27.645 INFO:teuthology.kill:2021-09-09 14:49:27,645.645 INFO:teuthology.orchestra.console:Power off for gibba041 completed 2021-09-09T14:49:27.775 INFO:teuthology.kill:2021-09-09 14:49:27,774.774 INFO:teuthology.lock.ops:unlocked gibba005.front.sepia.ceph.com 2021-09-09T14:49:27.783 INFO:teuthology.kill:2021-09-09 14:49:27,783.783 INFO:teuthology.lock.ops:unlocked gibba041.front.sepia.ceph.com 2021-09-09T14:51:28.058 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-09-09T14:51:28.166 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-09-09T14:51:28.167 INFO:teuthology.nuke:Checking targets against current locks 2021-09-09T14:51:28.287 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/scripts/dispatcher.py", line 33, in main teuthology.dispatcher.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/dispatcher/supervisor.py", line 59, in main verbose File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/dispatcher/supervisor.py", line 152, in run_job unlock_targets(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/dispatcher/supervisor.py", line 196, in unlock_targets nuke(fake_ctx, True) File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable