2021-06-22T19:18:35.963 INFO:root:teuthology version: 1.1.0-f359b10d 2021-06-22T19:18:36.076 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba004.front.sepia.ceph.com' reimaging 2021-06-22T19:18:36.077 INFO:teuthology.lock.ops:Updating [ubuntu@gibba004.front.sepia.ceph.com]: reset os type and version on server 2021-06-22T19:18:36.077 INFO:teuthology.lock.ops:Updating gibba004.front.sepia.ceph.com on lock server 2021-06-22T19:18:36.099 INFO:teuthology.lock.ops:Node 'ubuntu@gibba004.front.sepia.ceph.com' reimaging is complete 2021-06-22T19:18:36.099 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba011.front.sepia.ceph.com' reimaging 2021-06-22T19:18:36.100 INFO:teuthology.lock.ops:Updating [ubuntu@gibba011.front.sepia.ceph.com]: reset os type and version on server 2021-06-22T19:18:36.100 INFO:teuthology.lock.ops:Updating gibba011.front.sepia.ceph.com on lock server 2021-06-22T19:18:36.120 INFO:teuthology.lock.ops:Node 'ubuntu@gibba011.front.sepia.ceph.com' reimaging is complete 2021-06-22T19:18:36.293 INFO:teuthology.provision.fog.gibba004:Scheduling deploy of rhel 8.3 2021-06-22T19:18:36.324 INFO:teuthology.provision.fog.gibba011:Scheduling deploy of rhel 8.3 2021-06-22T19:18:36.656 INFO:teuthology.orchestra.console:Power off gibba004 2021-06-22T19:18:36.671 INFO:teuthology.orchestra.console:Power off gibba011 2021-06-22T19:18:36.814 INFO:teuthology.orchestra.console:Power off for gibba004 completed 2021-06-22T19:18:36.833 INFO:teuthology.orchestra.console:Power off for gibba011 completed 2021-06-22T19:18:36.916 INFO:teuthology.orchestra.console:Power on gibba004 2021-06-22T19:18:36.934 INFO:teuthology.orchestra.console:Power on gibba011 2021-06-22T19:18:41.199 INFO:teuthology.orchestra.console:Power on for gibba011 completed 2021-06-22T19:18:41.301 INFO:teuthology.provision.fog.gibba011:Waiting for deploy to finish 2021-06-22T19:18:45.296 INFO:teuthology.orchestra.console:Power on for gibba004 completed 2021-06-22T19:18:45.398 INFO:teuthology.provision.fog.gibba004:Waiting for deploy to finish 2021-06-22T19:20:46.530 ERROR:teuthology.orchestra.connection:Error authenticating with gibba004.front.sepia.ceph.com: Authentication failed. 2021-06-22T19:22:41.769 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-22T19:22:57.298 INFO:teuthology.provision.fog.gibba011:Node is ready 2021-06-22T19:22:57.312 INFO:teuthology.orchestra.run.gibba011.stdout:gibba011.front.sepia.ceph.com 2021-06-22T19:22:57.365 INFO:teuthology.orchestra.run.gibba011.stdout:172.21.2.111 gibba011.front.sepia.ceph.com gibba011 2021-06-22T19:22:57.708 INFO:teuthology.provision.fog.gibba011:Deploy complete! 2021-06-22T19:23:08.851 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-22T19:23:09.103 INFO:teuthology.provision.fog.gibba004:Node is ready 2021-06-22T19:23:09.117 INFO:teuthology.orchestra.run.gibba004.stdout:gibba004.front.sepia.ceph.com 2021-06-22T19:23:09.169 INFO:teuthology.orchestra.run.gibba004.stdout:172.21.2.104 gibba004.front.sepia.ceph.com gibba004 2021-06-22T19:23:09.462 INFO:teuthology.provision.fog.gibba004:Deploy complete! 2021-06-22T19:23:09.595 INFO:teuthology.lock.ops:Checking gibba004.front.sepia.ceph.com 2021-06-22T19:23:09.598 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-22T19:23:09.615 INFO:teuthology.lock.ops:Checking gibba011.front.sepia.ceph.com 2021-06-22T19:23:09.616 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-22T19:23:09.633 INFO:teuthology.lock.ops:Updating [gibba004.front.sepia.ceph.com]: set os type and version on server 2021-06-22T19:23:09.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-22T19:23:09.891 INFO:teuthology.orchestra.run.gibba004.stdout:x86_64 2021-06-22T19:23:09.943 INFO:teuthology.orchestra.run.gibba004.stdout:NAME="Red Hat Enterprise Linux" 2021-06-22T19:23:09.944 INFO:teuthology.orchestra.run.gibba004.stdout:VERSION="8.3 (Ootpa)" 2021-06-22T19:23:09.944 INFO:teuthology.orchestra.run.gibba004.stdout:ID="rhel" 2021-06-22T19:23:09.944 INFO:teuthology.orchestra.run.gibba004.stdout:ID_LIKE="fedora" 2021-06-22T19:23:09.945 INFO:teuthology.orchestra.run.gibba004.stdout:VERSION_ID="8.3" 2021-06-22T19:23:09.945 INFO:teuthology.orchestra.run.gibba004.stdout:PLATFORM_ID="platform:el8" 2021-06-22T19:23:09.945 INFO:teuthology.orchestra.run.gibba004.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-06-22T19:23:09.946 INFO:teuthology.orchestra.run.gibba004.stdout:ANSI_COLOR="0;31" 2021-06-22T19:23:09.946 INFO:teuthology.orchestra.run.gibba004.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-06-22T19:23:09.946 INFO:teuthology.orchestra.run.gibba004.stdout:HOME_URL="https://www.redhat.com/" 2021-06-22T19:23:09.946 INFO:teuthology.orchestra.run.gibba004.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-06-22T19:23:09.947 INFO:teuthology.orchestra.run.gibba004.stdout: 2021-06-22T19:23:09.947 INFO:teuthology.orchestra.run.gibba004.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-06-22T19:23:09.947 INFO:teuthology.orchestra.run.gibba004.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-06-22T19:23:09.948 INFO:teuthology.orchestra.run.gibba004.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-06-22T19:23:09.948 INFO:teuthology.orchestra.run.gibba004.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-06-22T19:23:09.949 INFO:teuthology.lock.ops:Updating gibba004.front.sepia.ceph.com on lock server 2021-06-22T19:23:09.965 INFO:teuthology.lock.ops:Updating [gibba011.front.sepia.ceph.com]: set os type and version on server 2021-06-22T19:23:09.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-22T19:23:10.258 INFO:teuthology.orchestra.run.gibba011.stdout:x86_64 2021-06-22T19:23:10.312 INFO:teuthology.orchestra.run.gibba011.stdout:NAME="Red Hat Enterprise Linux" 2021-06-22T19:23:10.312 INFO:teuthology.orchestra.run.gibba011.stdout:VERSION="8.3 (Ootpa)" 2021-06-22T19:23:10.312 INFO:teuthology.orchestra.run.gibba011.stdout:ID="rhel" 2021-06-22T19:23:10.312 INFO:teuthology.orchestra.run.gibba011.stdout:ID_LIKE="fedora" 2021-06-22T19:23:10.313 INFO:teuthology.orchestra.run.gibba011.stdout:VERSION_ID="8.3" 2021-06-22T19:23:10.313 INFO:teuthology.orchestra.run.gibba011.stdout:PLATFORM_ID="platform:el8" 2021-06-22T19:23:10.313 INFO:teuthology.orchestra.run.gibba011.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-06-22T19:23:10.313 INFO:teuthology.orchestra.run.gibba011.stdout:ANSI_COLOR="0;31" 2021-06-22T19:23:10.314 INFO:teuthology.orchestra.run.gibba011.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-06-22T19:23:10.314 INFO:teuthology.orchestra.run.gibba011.stdout:HOME_URL="https://www.redhat.com/" 2021-06-22T19:23:10.314 INFO:teuthology.orchestra.run.gibba011.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-06-22T19:23:10.314 INFO:teuthology.orchestra.run.gibba011.stdout: 2021-06-22T19:23:10.314 INFO:teuthology.orchestra.run.gibba011.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-06-22T19:23:10.315 INFO:teuthology.orchestra.run.gibba011.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-06-22T19:23:10.315 INFO:teuthology.orchestra.run.gibba011.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-06-22T19:23:10.315 INFO:teuthology.orchestra.run.gibba011.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-06-22T19:23:10.316 INFO:teuthology.lock.ops:Updating gibba011.front.sepia.ceph.com on lock server 2021-06-22T19:23:10.398 INFO:teuthology.dispatcher.supervisor:Running job 6142931 2021-06-22T19:23:10.398 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-05-30_07:01:02-rados-master-distro-basic-gibba/6142931 --name teuthology-2021-05-30_07:01:02-rados-master-distro-basic-gibba --description rados/upgrade/parallel/{0-start 1-tasks distro1$/{rhel_8.3_kubic_stable} mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} -- /home/teuthworker/archive/teuthology-2021-05-30_07:01:02-rados-master-distro-basic-gibba/6142931/orig.config.yaml 2021-06-22T19:23:10.404 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-05-30_07:01:02-rados-master-distro-basic-gibba/6142931 2021-06-22T19:23:10.405 INFO:teuthology.dispatcher.supervisor:Job PID: 14418 2021-06-22T19:23:10.405 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-06-23T01:03:21.064 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/teuthology-2021-05-30_07:01:02-rados-master-distro-basic-gibba/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2021-06-23T07:23:32.395 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-06-23T07:23:32.558 INFO:teuthology.kill:Killing Pids: {14418} 2021-06-23T07:23:33.575 INFO:teuthology.kill:Nuking machines: ['gibba004', 'gibba011'] 2021-06-23T07:23:34.479 INFO:teuthology.kill:2021-06-23 07:23:34,476.476 INFO:teuthology.nuke:targets: 2021-06-23T07:23:34.480 INFO:teuthology.kill: gibba004.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMjwcS2LtfAZ+ZB1k7HvMHVU4zUYZVQfhpPtDcOXL1VBu4I5LjIdMrX9sb16GCFBP/jCL/hfSfZnf3k6DiUbI8+vl1taEcBgreddXG4Q3OmVz7PyEk5gyTtACNQt0xvuuXWk5eSRuM5CGX4u2lovkkAo3aaHfREZZkBBU4J+qUGF4zmJEkK09HPz+f/jboPDQALXfhvDTiIelj14Gk+hPzf4Bibl33JawRRgMPWHUz0Ern4MDhIae3zEuITdkTX9sWCNlaPdB2By1hzUY7yF543FAzIsxE3kCUNg3jL/Q4FW8GDQCtmbuZSnADhi/ALrh+xAS5Y5R+lHpEewQvJzAdT9y+ghWwy7QbBRHwxIjE77iRqk9rpEthAMsDoqe6KPKBuAVS8B9uX+EjNHd2HLjVougIgsHM7R8rwgm5nZYP3J25lN8s6+jNQV4akgolLlwoPLsCWvfQVZxBH/6eX4lS6kNlLPrkvKQ1LIA9EGIpH+b1eF5cOdo0T0ID407Lwwk= 2021-06-23T07:23:34.480 INFO:teuthology.kill: gibba011.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCr800s1dTyS8snqK8e0hcIet9wvyTH6fprgjMu7NF1/7Mdb1QOZYzNiKiq77OKpvXfMiAO7VQd1fMOGY/0hN4Ib2tg5CFOs+z9DHd7xNbEwORZGyFPCsHCSg5kTTySpoL0JXMG1wU3s8LUyqcpLaSgfpxochEUaTvrENrCaRA6hXly2Kt3a2kX0tQOl0V7ZkT58wDLKfEWw+HcsSSfg3RwhX/KvLMmrd6QIh8HrgPYxl3xBh+YTTMhLwqR/N28eV/tvsv6yQ4EynID4sNIo6yL6zWBQgdsiFh9yCUR91YJoz5Tqd5/EJdy0grs1+ZgrwJpH0T9yXIadchmxt4hBIxyQ8gnetR15FvXgb+NG7f4bnVRhDyyWfiRlwW7OT2mrDB5+8yqknvg82ZA1iAPh98fomTbi0ILOCpdijp+MxZfKj5dlcORciIcw6gWZxyJxnKf409QOQUAltpI9baKpf3T/tFx5QBXaKtf1jMGmEnAyrW7sU772DSnQz+LEQpYjS0= 2021-06-23T07:23:34.481 INFO:teuthology.kill:2021-06-23 07:23:34,479.479 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-23T07:23:34.481 INFO:teuthology.kill:2021-06-23 07:23:34,481.481 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-23T07:23:34.538 INFO:teuthology.kill:2021-06-23 07:23:34,538.538 INFO:teuthology.orchestra.console:Power off gibba004 2021-06-23T07:23:34.551 INFO:teuthology.kill:2021-06-23 07:23:34,551.551 INFO:teuthology.orchestra.console:Power off gibba011 2021-06-23T07:23:42.919 INFO:teuthology.kill:2021-06-23 07:23:42,919.919 INFO:teuthology.orchestra.console:Power off for gibba011 completed 2021-06-23T07:23:47.024 INFO:teuthology.kill:2021-06-23 07:23:47,024.024 INFO:teuthology.orchestra.console:Power off for gibba004 completed 2021-06-23T07:23:47.234 INFO:teuthology.task.internal:roles: ubuntu@gibba004.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2021-06-23T07:23:47.234 INFO:teuthology.task.internal:roles: ubuntu@gibba011.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2021-06-23T07:23:47.235 INFO:teuthology.misc:Compressing logs... 2021-06-23T07:23:47.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-23T07:24:05.707 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-06-23T07:24:05.771 INFO:teuthology.kill:No teuthology processes running 2021-06-23T07:24:06.816 INFO:teuthology.kill:Nuking machines: ['gibba004', 'gibba011'] 2021-06-23T07:24:07.681 INFO:teuthology.kill:2021-06-23 07:24:07,680.680 INFO:teuthology.nuke:targets: 2021-06-23T07:24:07.682 INFO:teuthology.kill: gibba004.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMjwcS2LtfAZ+ZB1k7HvMHVU4zUYZVQfhpPtDcOXL1VBu4I5LjIdMrX9sb16GCFBP/jCL/hfSfZnf3k6DiUbI8+vl1taEcBgreddXG4Q3OmVz7PyEk5gyTtACNQt0xvuuXWk5eSRuM5CGX4u2lovkkAo3aaHfREZZkBBU4J+qUGF4zmJEkK09HPz+f/jboPDQALXfhvDTiIelj14Gk+hPzf4Bibl33JawRRgMPWHUz0Ern4MDhIae3zEuITdkTX9sWCNlaPdB2By1hzUY7yF543FAzIsxE3kCUNg3jL/Q4FW8GDQCtmbuZSnADhi/ALrh+xAS5Y5R+lHpEewQvJzAdT9y+ghWwy7QbBRHwxIjE77iRqk9rpEthAMsDoqe6KPKBuAVS8B9uX+EjNHd2HLjVougIgsHM7R8rwgm5nZYP3J25lN8s6+jNQV4akgolLlwoPLsCWvfQVZxBH/6eX4lS6kNlLPrkvKQ1LIA9EGIpH+b1eF5cOdo0T0ID407Lwwk= 2021-06-23T07:24:07.682 INFO:teuthology.kill: gibba011.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCr800s1dTyS8snqK8e0hcIet9wvyTH6fprgjMu7NF1/7Mdb1QOZYzNiKiq77OKpvXfMiAO7VQd1fMOGY/0hN4Ib2tg5CFOs+z9DHd7xNbEwORZGyFPCsHCSg5kTTySpoL0JXMG1wU3s8LUyqcpLaSgfpxochEUaTvrENrCaRA6hXly2Kt3a2kX0tQOl0V7ZkT58wDLKfEWw+HcsSSfg3RwhX/KvLMmrd6QIh8HrgPYxl3xBh+YTTMhLwqR/N28eV/tvsv6yQ4EynID4sNIo6yL6zWBQgdsiFh9yCUR91YJoz5Tqd5/EJdy0grs1+ZgrwJpH0T9yXIadchmxt4hBIxyQ8gnetR15FvXgb+NG7f4bnVRhDyyWfiRlwW7OT2mrDB5+8yqknvg82ZA1iAPh98fomTbi0ILOCpdijp+MxZfKj5dlcORciIcw6gWZxyJxnKf409QOQUAltpI9baKpf3T/tFx5QBXaKtf1jMGmEnAyrW7sU772DSnQz+LEQpYjS0= 2021-06-23T07:24:07.700 INFO:teuthology.kill:2021-06-23 07:24:07,700.700 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-23T07:24:07.703 INFO:teuthology.kill:2021-06-23 07:24:07,703.703 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-23T07:24:07.756 INFO:teuthology.kill:2021-06-23 07:24:07,756.756 INFO:teuthology.orchestra.console:Power off gibba011 2021-06-23T07:24:07.768 INFO:teuthology.kill:2021-06-23 07:24:07,768.768 INFO:teuthology.orchestra.console:Power off gibba004 2021-06-23T07:24:07.903 INFO:teuthology.kill:2021-06-23 07:24:07,903.903 INFO:teuthology.orchestra.console:Power off for gibba011 completed 2021-06-23T07:24:07.914 INFO:teuthology.kill:2021-06-23 07:24:07,914.914 INFO:teuthology.orchestra.console:Power off for gibba004 completed 2021-06-23T07:24:08.042 INFO:teuthology.kill:2021-06-23 07:24:08,041.041 INFO:teuthology.lock.ops:unlocked gibba011.front.sepia.ceph.com 2021-06-23T07:24:08.050 INFO:teuthology.kill:2021-06-23 07:24:08,050.050 ERROR:teuthology.lock.ops:failed to unlock gibba004.front.sepia.ceph.com. reason: 500 2021-06-23T07:26:08.216 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-06-23T07:26:08.311 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-06-23T07:26:08.311 INFO:teuthology.nuke:Checking targets against current locks 2021-06-23T07:26:08.380 INFO:teuthology.nuke:Not nuking gibba011.front.sepia.ceph.com because description doesn't match 2021-06-23T07:26:08.397 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-23T07:26:08.444 INFO:teuthology.orchestra.console:Power off gibba004 2021-06-23T07:26:08.592 INFO:teuthology.orchestra.console:Power off for gibba004 completed 2021-06-23T07:26:08.734 INFO:teuthology.lock.ops:unlocked gibba004.front.sepia.ceph.com