Internal Server Error
2021-08-19T19:29:44.415 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba037.front.sepia.ceph.com' reimaging 2021-08-19T19:29:44.416 INFO:teuthology.lock.ops:Updating [ubuntu@gibba037.front.sepia.ceph.com]: reset os type and version on server 2021-08-19T19:29:44.416 INFO:teuthology.lock.ops:Updating gibba037.front.sepia.ceph.com on lock server 2021-08-19T19:29:44.428 INFO:teuthology.lock.ops:Node 'ubuntu@gibba037.front.sepia.ceph.com' reimaging is complete 2021-08-19T19:29:44.625 INFO:teuthology.provision.fog.gibba037:Scheduling deploy of rhel 8.4 2021-08-19T19:29:44.897 INFO:teuthology.orchestra.console:Power off gibba037 2021-08-19T19:29:45.042 INFO:teuthology.orchestra.console:Power off for gibba037 completed 2021-08-19T19:29:45.143 INFO:teuthology.orchestra.console:Power on gibba037 2021-08-19T19:29:53.513 INFO:teuthology.orchestra.console:Power on for gibba037 completed 2021-08-19T19:29:53.615 INFO:teuthology.provision.fog.gibba037:Waiting for deploy to finish 2021-08-19T19:34:19.436 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-19T19:34:19.620 INFO:teuthology.provision.fog.gibba037:Node is ready 2021-08-19T19:34:19.630 INFO:teuthology.orchestra.run.gibba037.stdout:gibba037.front.sepia.ceph.com 2021-08-19T19:34:19.682 INFO:teuthology.orchestra.run.gibba037.stdout:172.21.2.137 gibba037.front.sepia.ceph.com gibba037 2021-08-19T19:34:19.975 INFO:teuthology.provision.fog.gibba037:Deploy complete! 2021-08-19T19:34:20.091 INFO:teuthology.lock.ops:Checking gibba037.front.sepia.ceph.com 2021-08-19T19:34:20.092 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-19T19:34:20.108 INFO:teuthology.lock.ops:Updating [gibba037.front.sepia.ceph.com]: set os type and version on server 2021-08-19T19:34:20.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-19T19:34:20.309 INFO:teuthology.orchestra.run.gibba037.stdout:x86_64 2021-08-19T19:34:20.361 INFO:teuthology.orchestra.run.gibba037.stdout:NAME="Red Hat Enterprise Linux" 2021-08-19T19:34:20.361 INFO:teuthology.orchestra.run.gibba037.stdout:VERSION="8.4 (Ootpa)" 2021-08-19T19:34:20.362 INFO:teuthology.orchestra.run.gibba037.stdout:ID="rhel" 2021-08-19T19:34:20.362 INFO:teuthology.orchestra.run.gibba037.stdout:ID_LIKE="fedora" 2021-08-19T19:34:20.362 INFO:teuthology.orchestra.run.gibba037.stdout:VERSION_ID="8.4" 2021-08-19T19:34:20.362 INFO:teuthology.orchestra.run.gibba037.stdout:PLATFORM_ID="platform:el8" 2021-08-19T19:34:20.363 INFO:teuthology.orchestra.run.gibba037.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2021-08-19T19:34:20.363 INFO:teuthology.orchestra.run.gibba037.stdout:ANSI_COLOR="0;31" 2021-08-19T19:34:20.363 INFO:teuthology.orchestra.run.gibba037.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2021-08-19T19:34:20.363 INFO:teuthology.orchestra.run.gibba037.stdout:HOME_URL="https://www.redhat.com/" 2021-08-19T19:34:20.363 INFO:teuthology.orchestra.run.gibba037.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2021-08-19T19:34:20.364 INFO:teuthology.orchestra.run.gibba037.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-08-19T19:34:20.364 INFO:teuthology.orchestra.run.gibba037.stdout: 2021-08-19T19:34:20.364 INFO:teuthology.orchestra.run.gibba037.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-08-19T19:34:20.364 INFO:teuthology.orchestra.run.gibba037.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2021-08-19T19:34:20.365 INFO:teuthology.orchestra.run.gibba037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-08-19T19:34:20.365 INFO:teuthology.orchestra.run.gibba037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2021-08-19T19:34:20.366 INFO:teuthology.lock.ops:Updating gibba037.front.sepia.ceph.com on lock server 2021-08-19T19:34:20.434 INFO:teuthology.dispatcher.supervisor:Running job 6347240 2021-08-19T19:34:20.434 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/bin/teuthology -v --owner scheduled_cbodley@teuthology --archive /home/teuthworker/archive/cbodley-2021-08-19_16:14:55-rgw-wip-cbodley-testing-distro-basic-gibba/6347240 --name cbodley-2021-08-19_16:14:55-rgw-wip-cbodley-testing-distro-basic-gibba --description rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} -- /home/teuthworker/archive/cbodley-2021-08-19_16:14:55-rgw-wip-cbodley-testing-distro-basic-gibba/6347240/orig.config.yaml 2021-08-19T19:34:20.437 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/cbodley-2021-08-19_16:14:55-rgw-wip-cbodley-testing-distro-basic-gibba/6347240 2021-08-19T19:34:20.438 INFO:teuthology.dispatcher.supervisor:Job PID: 4292 2021-08-19T19:34:20.438 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-08-20T01:38:43.733 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/cbodley-2021-08-19_16:14:55-rgw-wip-cbodley-testing-distro-basic-gibba/jobs/ failed with status 500:Internal Server Error
2021-08-20T07:35:08.019 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-08-20T07:35:08.231 INFO:teuthology.kill:Killing Pids: {4292} 2021-08-20T07:35:09.345 INFO:teuthology.kill:Nuking machines: ['gibba037'] 2021-08-20T07:35:10.380 INFO:teuthology.kill:2021-08-20 07:35:10,380.380 INFO:teuthology.nuke:targets: 2021-08-20T07:35:10.381 INFO:teuthology.kill: gibba037.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLX4e0DA4K7ZSEzd0h2iHgP0j+75Cd4tm9tT8I2g4J8BqFOQ/FmAcpxE/N06Em6eZ/2bs2vUPr/FTuGtSLlmStWMmyjAhQCwIAAW1trbgITWJHTq9C0jtuPKiXWNswFSRjPeESLcMSetILblgB1wRufMYg7uKRZvKxnyZq5/kxOPSaSHS/GXNiiHDT5UADiUvFK2IWYW5Y841uKnPbjFnIyydl8sy+4cAum3VDTfGiEEGyIjhB2V2kDsiKJrOQ+8FinSAhfmaAq/A0cQUcw0+wa/nFgWtQ5BBt3f2U25I/50zbuGtRYIAEpfcKHbWe9EJPxWI4B6nWKxyWzt3nJaD/OnW7G3AGrYJFakhKlQPTr209OodQhrgSoxY/UDsR/3ffNsaadb+3WwZ+ZnfBVCbHa6UK6Mffyho8yLwa0bep/iJSxerRkaeUv+ErJyz03Fi0OugdYqMzT6LmCCrwdwY2IImigBpSqI+eGpD6cTG6R3NV+2HVZUcTTCDZYGzR1kM= 2021-08-20T07:35:10.381 INFO:teuthology.kill:2021-08-20 07:35:10,380.380 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-20T07:35:10.436 INFO:teuthology.kill:2021-08-20 07:35:10,436.436 INFO:teuthology.orchestra.console:Power off gibba037 2021-08-20T07:35:22.924 INFO:teuthology.kill:2021-08-20 07:35:22,924.924 INFO:teuthology.orchestra.console:Power off for gibba037 completed 2021-08-20T07:35:23.161 INFO:teuthology.task.internal:roles: ubuntu@gibba037.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2021-08-20T07:35:23.162 INFO:teuthology.misc:Compressing logs... 2021-08-20T07:35:23.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-20T07:35:57.643 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/cluster.py", line 64, in