2021-03-28T15:13:10.624 INFO:root:teuthology version: 1.1.0-d5a60f86 2021-03-28T15:13:10.797 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi089.front.sepia.ceph.com' reimaging 2021-03-28T15:13:10.798 INFO:teuthology.lock.ops:Updating [ubuntu@smithi089.front.sepia.ceph.com]: reset os type and version on server 2021-03-28T15:13:10.798 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2021-03-28T15:13:10.815 INFO:teuthology.lock.ops:Node 'ubuntu@smithi089.front.sepia.ceph.com' reimaging is complete 2021-03-28T15:13:10.815 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi200.front.sepia.ceph.com' reimaging 2021-03-28T15:13:10.815 INFO:teuthology.lock.ops:Updating [ubuntu@smithi200.front.sepia.ceph.com]: reset os type and version on server 2021-03-28T15:13:10.815 INFO:teuthology.lock.ops:Updating smithi200.front.sepia.ceph.com on lock server 2021-03-28T15:13:10.831 INFO:teuthology.lock.ops:Node 'ubuntu@smithi200.front.sepia.ceph.com' reimaging is complete 2021-03-28T15:13:11.015 INFO:teuthology.provision.fog.smithi089:Scheduling deploy of rhel 8.3 2021-03-28T15:13:11.031 INFO:teuthology.provision.fog.smithi200:Scheduling deploy of rhel 8.3 2021-03-28T15:13:11.388 INFO:teuthology.orchestra.console:Power off smithi089 2021-03-28T15:13:11.424 INFO:teuthology.orchestra.console:Power off smithi200 2021-03-28T15:13:19.782 INFO:teuthology.orchestra.console:Power off for smithi089 completed 2021-03-28T15:13:19.789 INFO:teuthology.orchestra.console:Power off for smithi200 completed 2021-03-28T15:13:19.886 INFO:teuthology.orchestra.console:Power on smithi089 2021-03-28T15:13:19.896 INFO:teuthology.orchestra.console:Power on smithi200 2021-03-28T15:13:28.283 INFO:teuthology.orchestra.console:Power on for smithi200 completed 2021-03-28T15:13:28.315 INFO:teuthology.orchestra.console:Power on for smithi089 completed 2021-03-28T15:13:28.385 INFO:teuthology.provision.fog.smithi200:Waiting for deploy to finish 2021-03-28T15:13:28.419 INFO:teuthology.provision.fog.smithi089:Waiting for deploy to finish 2021-03-28T15:17:35.509 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-28T15:17:35.569 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-28T15:17:50.885 INFO:teuthology.provision.fog.smithi200:Node is ready 2021-03-28T15:17:50.904 INFO:teuthology.orchestra.run.smithi200.stdout:smithi087 2021-03-28T15:17:50.960 INFO:teuthology.orchestra.run.smithi200.stdout:172.21.15.87 smithi087.front.sepia.ceph.com smithi087 2021-03-28T15:17:51.306 INFO:teuthology.provision.fog.smithi200:Deploy complete! 2021-03-28T15:17:55.793 INFO:teuthology.provision.fog.smithi089:Node is ready 2021-03-28T15:17:55.813 INFO:teuthology.orchestra.run.smithi089.stdout:smithi089.front.sepia.ceph.com 2021-03-28T15:17:55.870 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89 smithi089.front.sepia.ceph.com smithi089 2021-03-28T15:17:56.239 INFO:teuthology.provision.fog.smithi089:Deploy complete! 2021-03-28T15:17:56.404 INFO:teuthology.lock.ops:Checking smithi089.front.sepia.ceph.com 2021-03-28T15:17:56.405 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-28T15:17:56.424 INFO:teuthology.lock.ops:Checking smithi200.front.sepia.ceph.com 2021-03-28T15:17:56.424 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-28T15:17:56.439 INFO:teuthology.lock.ops:Updating [smithi089.front.sepia.ceph.com]: set os type and version on server 2021-03-28T15:17:56.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-28T15:17:56.721 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2021-03-28T15:17:56.776 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="Red Hat Enterprise Linux" 2021-03-28T15:17:56.776 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="8.3 (Ootpa)" 2021-03-28T15:17:56.776 INFO:teuthology.orchestra.run.smithi089.stdout:ID="rhel" 2021-03-28T15:17:56.777 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE="fedora" 2021-03-28T15:17:56.777 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="8.3" 2021-03-28T15:17:56.777 INFO:teuthology.orchestra.run.smithi089.stdout:PLATFORM_ID="platform:el8" 2021-03-28T15:17:56.777 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-03-28T15:17:56.777 INFO:teuthology.orchestra.run.smithi089.stdout:ANSI_COLOR="0;31" 2021-03-28T15:17:56.777 INFO:teuthology.orchestra.run.smithi089.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-03-28T15:17:56.777 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://www.redhat.com/" 2021-03-28T15:17:56.777 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-03-28T15:17:56.777 INFO:teuthology.orchestra.run.smithi089.stdout: 2021-03-28T15:17:56.777 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-03-28T15:17:56.777 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-03-28T15:17:56.778 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-03-28T15:17:56.778 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-03-28T15:17:56.779 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2021-03-28T15:17:56.796 INFO:teuthology.lock.ops:Updating [smithi200.front.sepia.ceph.com]: set os type and version on server 2021-03-28T15:17:56.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-28T15:17:57.092 INFO:teuthology.orchestra.run.smithi200.stdout:x86_64 2021-03-28T15:17:57.145 INFO:teuthology.orchestra.run.smithi200.stdout:NAME="Red Hat Enterprise Linux" 2021-03-28T15:17:57.146 INFO:teuthology.orchestra.run.smithi200.stdout:VERSION="8.3 (Ootpa)" 2021-03-28T15:17:57.146 INFO:teuthology.orchestra.run.smithi200.stdout:ID="rhel" 2021-03-28T15:17:57.146 INFO:teuthology.orchestra.run.smithi200.stdout:ID_LIKE="fedora" 2021-03-28T15:17:57.146 INFO:teuthology.orchestra.run.smithi200.stdout:VERSION_ID="8.3" 2021-03-28T15:17:57.146 INFO:teuthology.orchestra.run.smithi200.stdout:PLATFORM_ID="platform:el8" 2021-03-28T15:17:57.146 INFO:teuthology.orchestra.run.smithi200.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-03-28T15:17:57.146 INFO:teuthology.orchestra.run.smithi200.stdout:ANSI_COLOR="0;31" 2021-03-28T15:17:57.146 INFO:teuthology.orchestra.run.smithi200.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-03-28T15:17:57.146 INFO:teuthology.orchestra.run.smithi200.stdout:HOME_URL="https://www.redhat.com/" 2021-03-28T15:17:57.146 INFO:teuthology.orchestra.run.smithi200.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-03-28T15:17:57.146 INFO:teuthology.orchestra.run.smithi200.stdout: 2021-03-28T15:17:57.147 INFO:teuthology.orchestra.run.smithi200.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-03-28T15:17:57.147 INFO:teuthology.orchestra.run.smithi200.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-03-28T15:17:57.147 INFO:teuthology.orchestra.run.smithi200.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-03-28T15:17:57.147 INFO:teuthology.orchestra.run.smithi200.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-03-28T15:17:57.148 INFO:teuthology.lock.ops:Updating smithi200.front.sepia.ceph.com on lock server 2021-03-28T15:17:57.235 INFO:teuthology.dispatcher.supervisor:Running job 6005502 2021-03-28T15:17:57.236 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-03-28_04:17:03-fs-pacific-distro-basic-smithi/6005502 --name teuthology-2021-03-28_04:17:03-fs-pacific-distro-basic-smithi --description fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} -- /home/teuthworker/archive/teuthology-2021-03-28_04:17:03-fs-pacific-distro-basic-smithi/6005502/orig.config.yaml 2021-03-28T15:17:57.242 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-03-28_04:17:03-fs-pacific-distro-basic-smithi/6005502 2021-03-28T15:17:57.242 INFO:teuthology.dispatcher.supervisor:Job PID: 26703 2021-03-28T15:17:57.242 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-03-28T17:05:13.108 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/teuthology-2021-03-28_04:17:03-fs-pacific-distro-basic-smithi/jobs/ failed with status 504: 504 Gateway Time-out

504 Gateway Time-out

nginx/1.10.3 (Ubuntu)
2021-03-29T03:18:47.059 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-29T03:18:47.373 INFO:teuthology.kill:Killing Pids: {26703} 2021-03-29T03:18:49.655 INFO:teuthology.kill:Nuking machines: ['smithi089', 'smithi200'] 2021-03-29T03:18:51.128 INFO:teuthology.kill:2021-03-29 03:18:51,128.128 INFO:teuthology.nuke:targets: 2021-03-29T03:18:51.129 INFO:teuthology.kill: smithi089.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDaT1rza0ahyhP8FIec8Zq3/d/A8Y4i2bDq6Qy8R0a66C7VyKBN2vx3j7ixW3cikBc6Trhd+XZYHLVlzh55rWrp4NXooerTCDty3QaqYltSyqoOJF/FhePYfeEHuLSV8NWTu9cXA42Vxtp8txt2Zl6mk4hPjXpT8EhcXp0DSgeZEik9KxNN3X+mDP6+wg/1WlFn98aoTPIkOvakpwCGWr6SSYLicR49+3IArP94B113Mbecwwb8/7xZZqQ4fNwDaQuZOt3KvVf0qEiDoglX7UYCLw7eLt9dWXZ6/h3bip7/xSQeThHJ54vEmP/RAcDq3JWXqWnhHVJJX+KtieL405p18vgH066woDu6A2jotaRbaife+c+iZiFDN0klVCju4JmacBsmL0ztTTJhTzSKIKDCdRp/ah37zC+0+AUdyX5S8HyeBrBweFbzo8h+8cNCvBAmKmhMKDeYqoYe0+qIXpE12KJBI3MQI6VJFws3r3F3ju97es8gn+2pPlnbsKgHoFs= 2021-03-29T03:18:51.129 INFO:teuthology.kill: smithi200.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC80ai0p+xJsdioxUOs0moCyvuY+lvUgKwBdIWbqHHGny27mwQTlGONP/k6fLAo/Pe0HNDSDLX5JS9QtvpPVv5hr9+NPvGmLN1PoTBdSkb7dZZ1ySILMmWmukARzbw8Y3Z5it/Ecv5rcX/IhbCRpgHkiDww/cw/TSecAdilMYWx4B8PWH/jxS9QaJzscYl8kmWRbfhq43br2p4EQddDYHT8Zy3QTBBeZSdp7u5A1y4034LpLt2p4ojux/tIcemhXoVCvWF+Zl/mWvjYLhzFuVjXrvDAYAMGVtWlw1Vh8X4kfF9LNsLTYM+nmzoJMX59fk9EXG6r7fq0/4mwyuq8YgD8WCNQtSZC6Di9ulXrXy+j2JzPVdSF01HNY8rDmplM/Zp6Iga7Fydy7TZiiYDoszOR63pEIjBmkPu4nZj1zI8OzhHeC1SXoJoOGl4UdDl3Yg4gLNRgcmXL0E7vqy7wgqMr2/LR+RxgvSVVsvsFlga1RrFeSGAwP12ViGQOrRGYpO0= 2021-03-29T03:18:51.129 INFO:teuthology.kill:2021-03-29 03:18:51,129.129 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-29T03:18:51.132 INFO:teuthology.kill:2021-03-29 03:18:51,131.131 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-29T03:18:51.219 INFO:teuthology.kill:2021-03-29 03:18:51,218.218 INFO:teuthology.orchestra.console:Power off smithi089 2021-03-29T03:18:51.238 INFO:teuthology.kill:2021-03-29 03:18:51,237.237 INFO:teuthology.orchestra.console:Power off smithi200 2021-03-29T03:19:03.718 INFO:teuthology.kill:2021-03-29 03:19:03,718.718 INFO:teuthology.orchestra.console:Power off for smithi089 completed 2021-03-29T03:19:03.723 INFO:teuthology.kill:2021-03-29 03:19:03,723.723 INFO:teuthology.orchestra.console:Power off for smithi200 completed 2021-03-29T03:19:04.009 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-03-29T03:19:04.010 INFO:teuthology.task.internal:roles: ubuntu@smithi200.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2021-03-29T03:19:04.010 INFO:teuthology.misc:Compressing logs... 2021-03-29T03:19:04.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-29T03:19:05.863 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-29T03:19:05.944 INFO:teuthology.kill:No teuthology processes running 2021-03-29T03:19:07.952 INFO:teuthology.kill:Nuking machines: ['smithi089', 'smithi200'] 2021-03-29T03:19:09.990 INFO:teuthology.kill:2021-03-29 03:19:09,989.989 INFO:teuthology.nuke:targets: 2021-03-29T03:19:09.990 INFO:teuthology.kill: smithi089.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDaT1rza0ahyhP8FIec8Zq3/d/A8Y4i2bDq6Qy8R0a66C7VyKBN2vx3j7ixW3cikBc6Trhd+XZYHLVlzh55rWrp4NXooerTCDty3QaqYltSyqoOJF/FhePYfeEHuLSV8NWTu9cXA42Vxtp8txt2Zl6mk4hPjXpT8EhcXp0DSgeZEik9KxNN3X+mDP6+wg/1WlFn98aoTPIkOvakpwCGWr6SSYLicR49+3IArP94B113Mbecwwb8/7xZZqQ4fNwDaQuZOt3KvVf0qEiDoglX7UYCLw7eLt9dWXZ6/h3bip7/xSQeThHJ54vEmP/RAcDq3JWXqWnhHVJJX+KtieL405p18vgH066woDu6A2jotaRbaife+c+iZiFDN0klVCju4JmacBsmL0ztTTJhTzSKIKDCdRp/ah37zC+0+AUdyX5S8HyeBrBweFbzo8h+8cNCvBAmKmhMKDeYqoYe0+qIXpE12KJBI3MQI6VJFws3r3F3ju97es8gn+2pPlnbsKgHoFs= 2021-03-29T03:19:09.991 INFO:teuthology.kill: smithi200.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC80ai0p+xJsdioxUOs0moCyvuY+lvUgKwBdIWbqHHGny27mwQTlGONP/k6fLAo/Pe0HNDSDLX5JS9QtvpPVv5hr9+NPvGmLN1PoTBdSkb7dZZ1ySILMmWmukARzbw8Y3Z5it/Ecv5rcX/IhbCRpgHkiDww/cw/TSecAdilMYWx4B8PWH/jxS9QaJzscYl8kmWRbfhq43br2p4EQddDYHT8Zy3QTBBeZSdp7u5A1y4034LpLt2p4ojux/tIcemhXoVCvWF+Zl/mWvjYLhzFuVjXrvDAYAMGVtWlw1Vh8X4kfF9LNsLTYM+nmzoJMX59fk9EXG6r7fq0/4mwyuq8YgD8WCNQtSZC6Di9ulXrXy+j2JzPVdSF01HNY8rDmplM/Zp6Iga7Fydy7TZiiYDoszOR63pEIjBmkPu4nZj1zI8OzhHeC1SXoJoOGl4UdDl3Yg4gLNRgcmXL0E7vqy7wgqMr2/LR+RxgvSVVsvsFlga1RrFeSGAwP12ViGQOrRGYpO0= 2021-03-29T03:19:10.014 INFO:teuthology.kill:2021-03-29 03:19:10,014.014 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-29T03:19:10.026 INFO:teuthology.kill:2021-03-29 03:19:10,026.026 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-29T03:19:10.102 INFO:teuthology.kill:2021-03-29 03:19:10,102.102 INFO:teuthology.orchestra.console:Power off smithi089 2021-03-29T03:19:10.117 INFO:teuthology.kill:2021-03-29 03:19:10,116.116 INFO:teuthology.orchestra.console:Power off smithi200 2021-03-29T03:19:10.251 INFO:teuthology.kill:2021-03-29 03:19:10,251.251 INFO:teuthology.orchestra.console:Power off for smithi089 completed 2021-03-29T03:19:10.264 INFO:teuthology.kill:2021-03-29 03:19:10,263.263 INFO:teuthology.orchestra.console:Power off for smithi200 completed 2021-03-29T03:19:10.400 INFO:teuthology.kill:2021-03-29 03:19:10,399.399 INFO:teuthology.lock.ops:unlocked smithi089.front.sepia.ceph.com 2021-03-29T03:19:10.415 INFO:teuthology.kill:2021-03-29 03:19:10,415.415 INFO:teuthology.lock.ops:unlocked smithi200.front.sepia.ceph.com 2021-03-29T03:21:10.729 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-03-29T03:21:10.830 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-03-29T03:21:10.830 INFO:teuthology.nuke:Checking targets against current locks 2021-03-29T03:21:10.932 INFO:teuthology.nuke:Not nuking smithi089.front.sepia.ceph.com because description doesn't match 2021-03-29T03:21:10.932 INFO:teuthology.nuke:Not nuking smithi200.front.sepia.ceph.com because description doesn't match