2021-04-18T08:55:19.634 INFO:root:teuthology version: 1.1.0-2713a3cd 2021-04-18T08:55:19.772 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi071.front.sepia.ceph.com' reimaging 2021-04-18T08:55:19.774 INFO:teuthology.lock.ops:Updating [ubuntu@smithi071.front.sepia.ceph.com]: reset os type and version on server 2021-04-18T08:55:19.774 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2021-04-18T08:55:19.795 INFO:teuthology.lock.ops:Node 'ubuntu@smithi071.front.sepia.ceph.com' reimaging is complete 2021-04-18T08:55:19.796 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi186.front.sepia.ceph.com' reimaging 2021-04-18T08:55:19.796 INFO:teuthology.lock.ops:Updating [ubuntu@smithi186.front.sepia.ceph.com]: reset os type and version on server 2021-04-18T08:55:19.797 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2021-04-18T08:55:19.817 INFO:teuthology.lock.ops:Node 'ubuntu@smithi186.front.sepia.ceph.com' reimaging is complete 2021-04-18T08:55:20.027 INFO:teuthology.provision.fog.smithi071:Scheduling deploy of rhel 8.3 2021-04-18T08:55:20.043 INFO:teuthology.provision.fog.smithi186:Scheduling deploy of rhel 8.3 2021-04-18T08:55:20.505 INFO:teuthology.orchestra.console:Power off smithi186 2021-04-18T08:55:20.526 INFO:teuthology.orchestra.console:Power off smithi071 2021-04-18T08:55:28.911 INFO:teuthology.orchestra.console:Power off for smithi186 completed 2021-04-18T08:55:29.012 INFO:teuthology.orchestra.console:Power on smithi186 2021-04-18T08:55:33.033 INFO:teuthology.orchestra.console:Power off for smithi071 completed 2021-04-18T08:55:33.135 INFO:teuthology.orchestra.console:Power on smithi071 2021-04-18T08:55:37.381 INFO:teuthology.orchestra.console:Power on for smithi186 completed 2021-04-18T08:55:37.389 INFO:teuthology.orchestra.console:Power on for smithi071 completed 2021-04-18T08:55:37.483 INFO:teuthology.provision.fog.smithi186:Waiting for deploy to finish 2021-04-18T08:55:37.490 INFO:teuthology.provision.fog.smithi071:Waiting for deploy to finish 2021-04-18T09:00:18.417 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-18T09:00:18.445 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-18T09:00:18.721 INFO:teuthology.provision.fog.smithi071:Node is ready 2021-04-18T09:00:18.725 INFO:teuthology.provision.fog.smithi186:Node is ready 2021-04-18T09:00:18.736 INFO:teuthology.orchestra.run.smithi071.stdout:smithi071.front.sepia.ceph.com 2021-04-18T09:00:18.741 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186.front.sepia.ceph.com 2021-04-18T09:00:18.791 INFO:teuthology.orchestra.run.smithi071.stdout:172.21.15.71 smithi071.front.sepia.ceph.com smithi071 2021-04-18T09:00:18.796 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.186 smithi186.front.sepia.ceph.com smithi186 2021-04-18T09:00:19.112 INFO:teuthology.provision.fog.smithi186:Deploy complete! 2021-04-18T09:00:19.141 INFO:teuthology.provision.fog.smithi071:Deploy complete! 2021-04-18T09:00:19.259 INFO:teuthology.lock.ops:Checking smithi071.front.sepia.ceph.com 2021-04-18T09:00:19.260 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-18T09:00:19.277 INFO:teuthology.lock.ops:Checking smithi186.front.sepia.ceph.com 2021-04-18T09:00:19.277 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-18T09:00:19.292 INFO:teuthology.lock.ops:Updating [smithi071.front.sepia.ceph.com]: set os type and version on server 2021-04-18T09:00:19.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-18T09:00:19.572 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2021-04-18T09:00:19.626 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="Red Hat Enterprise Linux" 2021-04-18T09:00:19.627 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="8.3 (Ootpa)" 2021-04-18T09:00:19.627 INFO:teuthology.orchestra.run.smithi071.stdout:ID="rhel" 2021-04-18T09:00:19.627 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="fedora" 2021-04-18T09:00:19.628 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="8.3" 2021-04-18T09:00:19.628 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el8" 2021-04-18T09:00:19.628 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-04-18T09:00:19.628 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2021-04-18T09:00:19.628 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-04-18T09:00:19.629 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://www.redhat.com/" 2021-04-18T09:00:19.629 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-04-18T09:00:19.629 INFO:teuthology.orchestra.run.smithi071.stdout: 2021-04-18T09:00:19.629 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-04-18T09:00:19.629 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-04-18T09:00:19.630 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-04-18T09:00:19.630 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-04-18T09:00:19.631 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2021-04-18T09:00:19.647 INFO:teuthology.lock.ops:Updating [smithi186.front.sepia.ceph.com]: set os type and version on server 2021-04-18T09:00:19.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-18T09:00:19.929 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2021-04-18T09:00:19.985 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="Red Hat Enterprise Linux" 2021-04-18T09:00:19.985 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="8.3 (Ootpa)" 2021-04-18T09:00:19.986 INFO:teuthology.orchestra.run.smithi186.stdout:ID="rhel" 2021-04-18T09:00:19.986 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="fedora" 2021-04-18T09:00:19.986 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="8.3" 2021-04-18T09:00:19.986 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el8" 2021-04-18T09:00:19.987 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-04-18T09:00:19.987 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2021-04-18T09:00:19.987 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-04-18T09:00:19.987 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://www.redhat.com/" 2021-04-18T09:00:19.988 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-04-18T09:00:19.988 INFO:teuthology.orchestra.run.smithi186.stdout: 2021-04-18T09:00:19.988 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-04-18T09:00:19.988 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-04-18T09:00:19.988 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-04-18T09:00:19.989 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-04-18T09:00:19.990 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2021-04-18T09:00:20.091 INFO:teuthology.dispatcher.supervisor:Running job 6054742 2021-04-18T09:00:20.092 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-04-18_04:17:03-fs-pacific-distro-basic-smithi/6054742 --name teuthology-2021-04-18_04:17:03-fs-pacific-distro-basic-smithi --description fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} -- /home/teuthworker/archive/teuthology-2021-04-18_04:17:03-fs-pacific-distro-basic-smithi/6054742/orig.config.yaml 2021-04-18T09:00:20.096 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-04-18_04:17:03-fs-pacific-distro-basic-smithi/6054742 2021-04-18T09:00:20.097 INFO:teuthology.dispatcher.supervisor:Job PID: 13118 2021-04-18T09:00:20.097 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-04-18T21:01:01.860 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-04-18T21:01:02.073 INFO:teuthology.kill:Killing Pids: {13118} 2021-04-18T21:01:03.147 INFO:teuthology.kill:Nuking machines: ['smithi071', 'smithi186'] 2021-04-18T21:01:04.078 INFO:teuthology.kill:2021-04-18 21:01:04,078.078 INFO:teuthology.nuke:targets: 2021-04-18T21:01:04.079 INFO:teuthology.kill: smithi071.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYBmb2KPVNSi/XR5lkXfXX1NBFFbEI5bRp/D/QhGp6Q7VegKiU1WeIOqBwo6oY4FIUi1o7dCQs6HNfGJZuAHtLqULYaE5a3h2yQPSVuW3fyIVg21lQiKYXkifMKj9XrfNfilAlPWRy21I9Z0xh2oj/XLy9Ms69xLEL4/FxfkT4frQBlAAYdBdIdkO8eQewgc5Jez4quK7UxvLU7q/lZN2gfnZCxGyQEOID6zc+E+zF+4nl+ZdosT50xtvNDYCFUFBO4Oi3SI2W1sPjTDvwOoA9/VRYyh6kVGqSIrWDjnigfb3+rueM75LvbmVuhQrFu3ExeWs2TL6Z1zX5uaiu8mztdeuvCX/PgDSkFwJBRmik6bTEuMw3cla7qctvxcYginbum7/nC3s13dADY5AvJxiq8mkeodHCOw81MVKxF7VjEC9TDbsTykkOPdPBf4phR1TwJVugtzidkQvggcelE5XU39wx6xf1eqbgPBhWn548MlgXnFSYZ6AB1gqKHyZd4yE= 2021-04-18T21:01:04.080 INFO:teuthology.kill: smithi186.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCr+jOXPNG43Lg9uWrMXIka7EtBupApcxFLk8PjE6zC+2/1CwiCpwm3i6kSwI5JZ9csEc8NP1sWCx7HF2jdd8Kx4V/dGUy+bJ7FGXYWb6wRVE12Py6Mw+BUzzbF1sqmnjl4m1gMm8oPqbCozLxxBspXDOg9YjUtyhn7VWu5m5gJZuy97LUQmFNQQsDcXUe9hZXj0lo80Y9Poj4H06iL8q49B0fRwF3GMeDtneqUoB+1tZpTTpgcRS7rEKoxlguNy1XKw0+t4+7+eUT3Wax9k7N6o1Jkemma0IgiI3s9JO6xvR7nuajnC6GDssjOo6xpnXShW3zGAt4Rl+OKvdThtb2tnLWn9rVJqx94eFbQCmHNUkAb3k34mpvQCzLsgWjDGACa1tEkDz8ks8DcjRpJ26bOnymFd7D07t0RgwKQfEeCC0AWQ4aQhrzaLyG82mN/JcOfccGCgc/nQDGG+U2fLnLFhLIJZ4hHlZ7xlidnV+wqj5+BE1YQ5bddF0j4GrNoxQk= 2021-04-18T21:01:04.080 INFO:teuthology.kill:2021-04-18 21:01:04,078.078 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-18T21:01:04.081 INFO:teuthology.kill:2021-04-18 21:01:04,081.081 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-18T21:01:04.153 INFO:teuthology.kill:2021-04-18 21:01:04,153.153 INFO:teuthology.orchestra.console:Power off smithi071 2021-04-18T21:01:04.167 INFO:teuthology.kill:2021-04-18 21:01:04,167.167 INFO:teuthology.orchestra.console:Power off smithi186 2021-04-18T21:01:12.516 INFO:teuthology.kill:2021-04-18 21:01:12,515.515 INFO:teuthology.orchestra.console:Power off for smithi071 completed 2021-04-18T21:01:16.631 INFO:teuthology.kill:2021-04-18 21:01:16,630.630 INFO:teuthology.orchestra.console:Power off for smithi186 completed 2021-04-18T21:01:16.849 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-04-18T21:01:16.850 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2021-04-18T21:01:16.850 INFO:teuthology.misc:Compressing logs... 2021-04-18T21:01:16.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-18T21:02:16.912 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-04-18T21:02:16.991 INFO:teuthology.kill:No teuthology processes running 2021-04-18T21:02:18.000 INFO:teuthology.kill:Nuking machines: ['smithi071', 'smithi186'] 2021-04-18T21:02:18.905 INFO:teuthology.kill:2021-04-18 21:02:18,904.904 INFO:teuthology.nuke:targets: 2021-04-18T21:02:18.905 INFO:teuthology.kill: smithi071.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYBmb2KPVNSi/XR5lkXfXX1NBFFbEI5bRp/D/QhGp6Q7VegKiU1WeIOqBwo6oY4FIUi1o7dCQs6HNfGJZuAHtLqULYaE5a3h2yQPSVuW3fyIVg21lQiKYXkifMKj9XrfNfilAlPWRy21I9Z0xh2oj/XLy9Ms69xLEL4/FxfkT4frQBlAAYdBdIdkO8eQewgc5Jez4quK7UxvLU7q/lZN2gfnZCxGyQEOID6zc+E+zF+4nl+ZdosT50xtvNDYCFUFBO4Oi3SI2W1sPjTDvwOoA9/VRYyh6kVGqSIrWDjnigfb3+rueM75LvbmVuhQrFu3ExeWs2TL6Z1zX5uaiu8mztdeuvCX/PgDSkFwJBRmik6bTEuMw3cla7qctvxcYginbum7/nC3s13dADY5AvJxiq8mkeodHCOw81MVKxF7VjEC9TDbsTykkOPdPBf4phR1TwJVugtzidkQvggcelE5XU39wx6xf1eqbgPBhWn548MlgXnFSYZ6AB1gqKHyZd4yE= 2021-04-18T21:02:18.906 INFO:teuthology.kill: smithi186.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCr+jOXPNG43Lg9uWrMXIka7EtBupApcxFLk8PjE6zC+2/1CwiCpwm3i6kSwI5JZ9csEc8NP1sWCx7HF2jdd8Kx4V/dGUy+bJ7FGXYWb6wRVE12Py6Mw+BUzzbF1sqmnjl4m1gMm8oPqbCozLxxBspXDOg9YjUtyhn7VWu5m5gJZuy97LUQmFNQQsDcXUe9hZXj0lo80Y9Poj4H06iL8q49B0fRwF3GMeDtneqUoB+1tZpTTpgcRS7rEKoxlguNy1XKw0+t4+7+eUT3Wax9k7N6o1Jkemma0IgiI3s9JO6xvR7nuajnC6GDssjOo6xpnXShW3zGAt4Rl+OKvdThtb2tnLWn9rVJqx94eFbQCmHNUkAb3k34mpvQCzLsgWjDGACa1tEkDz8ks8DcjRpJ26bOnymFd7D07t0RgwKQfEeCC0AWQ4aQhrzaLyG82mN/JcOfccGCgc/nQDGG+U2fLnLFhLIJZ4hHlZ7xlidnV+wqj5+BE1YQ5bddF0j4GrNoxQk= 2021-04-18T21:02:18.924 INFO:teuthology.kill:2021-04-18 21:02:18,924.924 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-18T21:02:18.936 INFO:teuthology.kill:2021-04-18 21:02:18,936.936 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-18T21:02:18.997 INFO:teuthology.kill:2021-04-18 21:02:18,997.997 INFO:teuthology.orchestra.console:Power off smithi071 2021-04-18T21:02:19.010 INFO:teuthology.kill:2021-04-18 21:02:19,010.010 INFO:teuthology.orchestra.console:Power off smithi186 2021-04-18T21:02:19.139 INFO:teuthology.kill:2021-04-18 21:02:19,138.138 INFO:teuthology.orchestra.console:Power off for smithi071 completed 2021-04-18T21:02:19.148 INFO:teuthology.kill:2021-04-18 21:02:19,148.148 INFO:teuthology.orchestra.console:Power off for smithi186 completed 2021-04-18T21:02:19.288 INFO:teuthology.kill:2021-04-18 21:02:19,287.287 INFO:teuthology.lock.ops:unlocked smithi071.front.sepia.ceph.com 2021-04-18T21:02:19.305 INFO:teuthology.kill:2021-04-18 21:02:19,305.305 INFO:teuthology.lock.ops:unlocked smithi186.front.sepia.ceph.com 2021-04-18T21:04:19.571 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15