2021-03-28T15:03:10.842 INFO:root:teuthology version: 1.1.0-d5a60f86 2021-03-28T15:03:10.977 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi061.front.sepia.ceph.com' reimaging 2021-03-28T15:03:10.978 INFO:teuthology.lock.ops:Updating [ubuntu@smithi061.front.sepia.ceph.com]: reset os type and version on server 2021-03-28T15:03:10.978 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server 2021-03-28T15:03:10.993 INFO:teuthology.lock.ops:Node 'ubuntu@smithi061.front.sepia.ceph.com' reimaging is complete 2021-03-28T15:03:10.993 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi084.front.sepia.ceph.com' reimaging 2021-03-28T15:03:10.993 INFO:teuthology.lock.ops:Updating [ubuntu@smithi084.front.sepia.ceph.com]: reset os type and version on server 2021-03-28T15:03:10.993 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2021-03-28T15:03:11.012 INFO:teuthology.lock.ops:Node 'ubuntu@smithi084.front.sepia.ceph.com' reimaging is complete 2021-03-28T15:03:11.278 INFO:teuthology.provision.fog.smithi061:Scheduling deploy of rhel 8.3 2021-03-28T15:03:11.281 INFO:teuthology.provision.fog.smithi084:Scheduling deploy of rhel 8.3 2021-03-28T15:03:11.778 INFO:teuthology.orchestra.console:Power off smithi061 2021-03-28T15:03:11.848 INFO:teuthology.orchestra.console:Power off smithi084 2021-03-28T15:03:20.151 INFO:teuthology.orchestra.console:Power off for smithi061 completed 2021-03-28T15:03:20.252 INFO:teuthology.orchestra.console:Power on smithi061 2021-03-28T15:03:24.320 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2021-03-28T15:03:24.421 INFO:teuthology.orchestra.console:Power on smithi084 2021-03-28T15:03:24.657 INFO:teuthology.orchestra.console:Power on for smithi061 completed 2021-03-28T15:03:24.758 INFO:teuthology.provision.fog.smithi061:Waiting for deploy to finish 2021-03-28T15:03:28.675 INFO:teuthology.orchestra.console:Power on for smithi084 completed 2021-03-28T15:03:28.777 INFO:teuthology.provision.fog.smithi084:Waiting for deploy to finish 2021-03-28T15:07:35.658 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-28T15:07:46.687 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-28T15:07:56.164 INFO:teuthology.provision.fog.smithi084:Node is ready 2021-03-28T15:07:56.183 INFO:teuthology.orchestra.run.smithi084.stdout:smithi084.front.sepia.ceph.com 2021-03-28T15:07:56.239 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84 smithi084.front.sepia.ceph.com smithi084 2021-03-28T15:07:56.602 INFO:teuthology.provision.fog.smithi084:Deploy complete! 2021-03-28T15:08:07.409 INFO:teuthology.provision.fog.smithi061:Node is ready 2021-03-28T15:08:07.428 INFO:teuthology.orchestra.run.smithi061.stdout:smithi061.front.sepia.ceph.com 2021-03-28T15:08:07.486 INFO:teuthology.orchestra.run.smithi061.stdout:172.21.15.61 smithi061.front.sepia.ceph.com smithi061 2021-03-28T15:08:07.858 INFO:teuthology.provision.fog.smithi061:Deploy complete! 2021-03-28T15:08:07.996 INFO:teuthology.lock.ops:Checking smithi061.front.sepia.ceph.com 2021-03-28T15:08:07.997 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-28T15:08:08.012 INFO:teuthology.lock.ops:Checking smithi084.front.sepia.ceph.com 2021-03-28T15:08:08.012 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-28T15:08:08.025 INFO:teuthology.lock.ops:Updating [smithi061.front.sepia.ceph.com]: set os type and version on server 2021-03-28T15:08:08.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-28T15:08:08.309 INFO:teuthology.orchestra.run.smithi061.stdout:x86_64 2021-03-28T15:08:08.363 INFO:teuthology.orchestra.run.smithi061.stdout:NAME="Red Hat Enterprise Linux" 2021-03-28T15:08:08.364 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION="8.3 (Ootpa)" 2021-03-28T15:08:08.364 INFO:teuthology.orchestra.run.smithi061.stdout:ID="rhel" 2021-03-28T15:08:08.364 INFO:teuthology.orchestra.run.smithi061.stdout:ID_LIKE="fedora" 2021-03-28T15:08:08.364 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_ID="8.3" 2021-03-28T15:08:08.364 INFO:teuthology.orchestra.run.smithi061.stdout:PLATFORM_ID="platform:el8" 2021-03-28T15:08:08.365 INFO:teuthology.orchestra.run.smithi061.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-03-28T15:08:08.365 INFO:teuthology.orchestra.run.smithi061.stdout:ANSI_COLOR="0;31" 2021-03-28T15:08:08.365 INFO:teuthology.orchestra.run.smithi061.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-03-28T15:08:08.365 INFO:teuthology.orchestra.run.smithi061.stdout:HOME_URL="https://www.redhat.com/" 2021-03-28T15:08:08.365 INFO:teuthology.orchestra.run.smithi061.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-03-28T15:08:08.365 INFO:teuthology.orchestra.run.smithi061.stdout: 2021-03-28T15:08:08.366 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-03-28T15:08:08.366 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-03-28T15:08:08.367 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-03-28T15:08:08.367 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-03-28T15:08:08.369 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server 2021-03-28T15:08:08.385 INFO:teuthology.lock.ops:Updating [smithi084.front.sepia.ceph.com]: set os type and version on server 2021-03-28T15:08:08.385 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-28T15:08:08.673 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2021-03-28T15:08:08.728 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="Red Hat Enterprise Linux" 2021-03-28T15:08:08.831 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="8.3 (Ootpa)" 2021-03-28T15:08:08.831 INFO:teuthology.orchestra.run.smithi084.stdout:ID="rhel" 2021-03-28T15:08:08.831 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="fedora" 2021-03-28T15:08:08.831 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="8.3" 2021-03-28T15:08:08.831 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el8" 2021-03-28T15:08:08.831 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-03-28T15:08:08.831 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2021-03-28T15:08:08.831 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-03-28T15:08:08.832 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://www.redhat.com/" 2021-03-28T15:08:08.832 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-03-28T15:08:08.832 INFO:teuthology.orchestra.run.smithi084.stdout: 2021-03-28T15:08:08.832 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-03-28T15:08:08.832 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-03-28T15:08:08.832 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-03-28T15:08:08.832 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-03-28T15:08:08.833 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2021-03-28T15:08:08.919 INFO:teuthology.dispatcher.supervisor:Running job 6005481 2021-03-28T15:08:08.919 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-03-28_04:17:03-fs-pacific-distro-basic-smithi/6005481 --name teuthology-2021-03-28_04:17:03-fs-pacific-distro-basic-smithi --description fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} -- /home/teuthworker/archive/teuthology-2021-03-28_04:17:03-fs-pacific-distro-basic-smithi/6005481/orig.config.yaml 2021-03-28T15:08:08.922 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-03-28_04:17:03-fs-pacific-distro-basic-smithi/6005481 2021-03-28T15:08:08.922 INFO:teuthology.dispatcher.supervisor:Job PID: 22121 2021-03-28T15:08:08.922 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-03-28T17:05:13.131 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/teuthology-2021-03-28_04:17:03-fs-pacific-distro-basic-smithi/jobs/ failed with status 504: 504 Gateway Time-out

504 Gateway Time-out

nginx/1.10.3 (Ubuntu)
2021-03-29T03:08:46.256 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-29T03:08:46.473 INFO:teuthology.kill:Killing Pids: {22121} 2021-03-29T03:09:01.411 INFO:teuthology.kill:Nuking machines: ['smithi061', 'smithi084'] 2021-03-29T03:09:03.321 INFO:teuthology.kill:2021-03-29 03:09:03,320.320 INFO:teuthology.nuke:targets: 2021-03-29T03:09:03.321 INFO:teuthology.kill: smithi061.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvq1B+73Y9Dj7bgSOUbxZAKbjOVgZy/uomAQZwhMQsgEarJMO9e3KFYsu9BQk1qEN9O4HRMalS0lvmcK0fFQOqTvu4lQV4BMKL1/yBK0s6YMbEqWKikbiHJTzSlJRkQB5Cq1XOm0gAoDFVRkLFXsBOP+IhJ3Ges6x2sv8UopJcjwIxKjV7WJJsjSpo3oYx+B/DsHCD7xm1poumwHIc6ZIhXlLXcpvrKyzdDuqog6eP1b1akFMp2A/2/ok+qTyw7mANsoVqT2jCIkv7L0ywbxwZBJWz++Gd5tjAa46WkIsWVIKWRY6TPQYqMRxyehHrtSdRyw4nf8bFNWNLDESLyehHM5ktfFTA0j19LDvrXH+b/yHWFBVokwamQUXpFlRmfeHBAwkKl7UMRjpatzRV3rWiW3Skba++SPnb8NPFJAwr4ElXyUNqDLpAORTBA188zkJiJ8zJRm0pSiPnbyqiQfsBfh5Zv7qJzklOtwbdGUUbNtQj6GLPmuZed0La6Gh30f8= 2021-03-29T03:09:03.321 INFO:teuthology.kill: smithi084.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1F2vyRALkelJdeiQvXdBAnUtcbUG62HujobDN6txUcgJU/ajnhGyYbKb3VNEgDNOQlf9hJ2BEWhv50JqibH4oDwiwGorAzkuijL+CjAXID6wB04A13P8rlY4w+WzhSpPQY93Q9lx67hD47MaP2fAE/LCzKuUhnMXhjMqGuhTLDtLQOaSwBSzBk1jO1A1BYujA7gXIQRA/DkvZIhuUC/yy+Npgcd+CIVq6vDX91XRKFRh2Oig5axNN61rn2MyamQ8p7+lZUWd/mHKIZbxrcSye+/fLuyZnGgRqRLJFQUs8YItE050Cw/CkvRfH/SGNq9rWHtXPuTuGOuKFzb3GGaP616ehcXFkTEaKx1Xexj5eoC4HzXbhJDci3wcy1XaY4jWWvi87sf4E2O+Cmrqbbfn8F8IWhHVntFqKWv7OgeKSZmydMXZAeGMhgfWYbVsqs08eIGRIbV4y+rO2n/ZlV1c3g163p51/DtSCJsGx4ETinzwgB62Lg85axbdCpbgjwr8= 2021-03-29T03:09:03.321 INFO:teuthology.kill:2021-03-29 03:09:03,321.321 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-29T03:09:03.325 INFO:teuthology.kill:2021-03-29 03:09:03,324.324 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-29T03:09:03.401 INFO:teuthology.kill:2021-03-29 03:09:03,400.400 INFO:teuthology.orchestra.console:Power off smithi061 2021-03-29T03:09:03.417 INFO:teuthology.kill:2021-03-29 03:09:03,417.417 INFO:teuthology.orchestra.console:Power off smithi084 2021-03-29T03:09:11.800 INFO:teuthology.kill:2021-03-29 03:09:11,799.799 INFO:teuthology.orchestra.console:Power off for smithi061 completed 2021-03-29T03:09:11.805 INFO:teuthology.kill:2021-03-29 03:09:11,805.805 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2021-03-29T03:09:12.088 INFO:teuthology.task.internal:roles: ubuntu@smithi061.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-03-29T03:09:12.088 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2021-03-29T03:09:12.088 INFO:teuthology.misc:Compressing logs... 2021-03-29T03:09:12.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-29T03:09:30.635 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-29T03:09:30.705 INFO:teuthology.kill:No teuthology processes running 2021-03-29T03:09:31.841 INFO:teuthology.kill:Nuking machines: ['smithi061', 'smithi084'] 2021-03-29T03:09:32.865 INFO:teuthology.kill:2021-03-29 03:09:32,864.864 INFO:teuthology.nuke:targets: 2021-03-29T03:09:32.865 INFO:teuthology.kill: smithi061.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvq1B+73Y9Dj7bgSOUbxZAKbjOVgZy/uomAQZwhMQsgEarJMO9e3KFYsu9BQk1qEN9O4HRMalS0lvmcK0fFQOqTvu4lQV4BMKL1/yBK0s6YMbEqWKikbiHJTzSlJRkQB5Cq1XOm0gAoDFVRkLFXsBOP+IhJ3Ges6x2sv8UopJcjwIxKjV7WJJsjSpo3oYx+B/DsHCD7xm1poumwHIc6ZIhXlLXcpvrKyzdDuqog6eP1b1akFMp2A/2/ok+qTyw7mANsoVqT2jCIkv7L0ywbxwZBJWz++Gd5tjAa46WkIsWVIKWRY6TPQYqMRxyehHrtSdRyw4nf8bFNWNLDESLyehHM5ktfFTA0j19LDvrXH+b/yHWFBVokwamQUXpFlRmfeHBAwkKl7UMRjpatzRV3rWiW3Skba++SPnb8NPFJAwr4ElXyUNqDLpAORTBA188zkJiJ8zJRm0pSiPnbyqiQfsBfh5Zv7qJzklOtwbdGUUbNtQj6GLPmuZed0La6Gh30f8= 2021-03-29T03:09:32.865 INFO:teuthology.kill: smithi084.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1F2vyRALkelJdeiQvXdBAnUtcbUG62HujobDN6txUcgJU/ajnhGyYbKb3VNEgDNOQlf9hJ2BEWhv50JqibH4oDwiwGorAzkuijL+CjAXID6wB04A13P8rlY4w+WzhSpPQY93Q9lx67hD47MaP2fAE/LCzKuUhnMXhjMqGuhTLDtLQOaSwBSzBk1jO1A1BYujA7gXIQRA/DkvZIhuUC/yy+Npgcd+CIVq6vDX91XRKFRh2Oig5axNN61rn2MyamQ8p7+lZUWd/mHKIZbxrcSye+/fLuyZnGgRqRLJFQUs8YItE050Cw/CkvRfH/SGNq9rWHtXPuTuGOuKFzb3GGaP616ehcXFkTEaKx1Xexj5eoC4HzXbhJDci3wcy1XaY4jWWvi87sf4E2O+Cmrqbbfn8F8IWhHVntFqKWv7OgeKSZmydMXZAeGMhgfWYbVsqs08eIGRIbV4y+rO2n/ZlV1c3g163p51/DtSCJsGx4ETinzwgB62Lg85axbdCpbgjwr8= 2021-03-29T03:09:32.886 INFO:teuthology.kill:2021-03-29 03:09:32,886.886 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-29T03:09:32.898 INFO:teuthology.kill:2021-03-29 03:09:32,898.898 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-29T03:09:33.011 INFO:teuthology.kill:2021-03-29 03:09:33,011.011 INFO:teuthology.orchestra.console:Power off smithi061 2021-03-29T03:09:33.023 INFO:teuthology.kill:2021-03-29 03:09:33,023.023 INFO:teuthology.orchestra.console:Power off smithi084 2021-03-29T03:09:33.151 INFO:teuthology.kill:2021-03-29 03:09:33,150.150 INFO:teuthology.orchestra.console:Power off for smithi061 completed 2021-03-29T03:09:33.163 INFO:teuthology.kill:2021-03-29 03:09:33,163.163 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2021-03-29T03:09:33.300 INFO:teuthology.kill:2021-03-29 03:09:33,299.299 INFO:teuthology.lock.ops:unlocked smithi061.front.sepia.ceph.com 2021-03-29T03:09:33.317 INFO:teuthology.kill:2021-03-29 03:09:33,316.316 INFO:teuthology.lock.ops:unlocked smithi084.front.sepia.ceph.com 2021-03-29T03:11:33.609 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-03-29T03:11:33.739 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-03-29T03:11:33.739 INFO:teuthology.nuke:Checking targets against current locks 2021-03-29T03:11:33.812 INFO:teuthology.nuke:Not nuking smithi061.front.sepia.ceph.com because description doesn't match 2021-03-29T03:11:33.812 INFO:teuthology.nuke:Not nuking smithi084.front.sepia.ceph.com because description doesn't match