2021-04-11T07:21:35.509 INFO:root:teuthology version: 1.1.0-85d61eae 2021-04-11T07:21:35.610 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi159.front.sepia.ceph.com' reimaging 2021-04-11T07:21:35.611 INFO:teuthology.lock.ops:Updating [ubuntu@smithi159.front.sepia.ceph.com]: reset os type and version on server 2021-04-11T07:21:35.612 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2021-04-11T07:21:35.643 INFO:teuthology.lock.ops:Node 'ubuntu@smithi159.front.sepia.ceph.com' reimaging is complete 2021-04-11T07:21:35.644 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi201.front.sepia.ceph.com' reimaging 2021-04-11T07:21:35.644 INFO:teuthology.lock.ops:Updating [ubuntu@smithi201.front.sepia.ceph.com]: reset os type and version on server 2021-04-11T07:21:35.645 INFO:teuthology.lock.ops:Updating smithi201.front.sepia.ceph.com on lock server 2021-04-11T07:21:35.672 INFO:teuthology.lock.ops:Node 'ubuntu@smithi201.front.sepia.ceph.com' reimaging is complete 2021-04-11T07:21:35.853 INFO:teuthology.provision.fog.smithi159:Scheduling deploy of rhel 8.3 2021-04-11T07:21:35.889 INFO:teuthology.provision.fog.smithi201:Scheduling deploy of rhel 8.3 2021-04-11T07:21:36.281 INFO:teuthology.orchestra.console:Power off smithi159 2021-04-11T07:21:36.345 INFO:teuthology.orchestra.console:Power off smithi201 2021-04-11T07:21:44.657 INFO:teuthology.orchestra.console:Power off for smithi159 completed 2021-04-11T07:21:44.709 INFO:teuthology.orchestra.console:Power off for smithi201 completed 2021-04-11T07:21:44.759 INFO:teuthology.orchestra.console:Power on smithi159 2021-04-11T07:21:44.810 INFO:teuthology.orchestra.console:Power on smithi201 2021-04-11T07:21:53.135 INFO:teuthology.orchestra.console:Power on for smithi159 completed 2021-04-11T07:21:53.171 INFO:teuthology.orchestra.console:Power on for smithi201 completed 2021-04-11T07:21:53.236 INFO:teuthology.provision.fog.smithi159:Waiting for deploy to finish 2021-04-11T07:21:53.274 INFO:teuthology.provision.fog.smithi201:Waiting for deploy to finish 2021-04-11T07:26:28.097 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-11T07:26:28.403 INFO:teuthology.provision.fog.smithi201:Node is ready 2021-04-11T07:26:28.418 INFO:teuthology.orchestra.run.smithi201.stdout:smithi201.front.sepia.ceph.com 2021-04-11T07:26:28.472 INFO:teuthology.orchestra.run.smithi201.stdout:172.21.15.201 smithi201.front.sepia.ceph.com smithi201 2021-04-11T07:26:28.794 INFO:teuthology.provision.fog.smithi201:Deploy complete! 2021-04-11T07:26:52.009 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-11T07:26:52.287 INFO:teuthology.provision.fog.smithi159:Node is ready 2021-04-11T07:26:52.302 INFO:teuthology.orchestra.run.smithi159.stdout:smithi159.front.sepia.ceph.com 2021-04-11T07:26:52.356 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.15.159 smithi159.front.sepia.ceph.com smithi159 2021-04-11T07:26:52.683 INFO:teuthology.provision.fog.smithi159:Deploy complete! 2021-04-11T07:26:52.818 INFO:teuthology.lock.ops:Checking smithi159.front.sepia.ceph.com 2021-04-11T07:26:52.820 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-11T07:26:52.839 INFO:teuthology.lock.ops:Checking smithi201.front.sepia.ceph.com 2021-04-11T07:26:52.839 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-11T07:26:52.859 INFO:teuthology.lock.ops:Updating [smithi159.front.sepia.ceph.com]: set os type and version on server 2021-04-11T07:26:52.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-11T07:26:53.145 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64 2021-04-11T07:26:53.200 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="Red Hat Enterprise Linux" 2021-04-11T07:26:53.200 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="8.3 (Ootpa)" 2021-04-11T07:26:53.200 INFO:teuthology.orchestra.run.smithi159.stdout:ID="rhel" 2021-04-11T07:26:53.201 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE="fedora" 2021-04-11T07:26:53.201 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="8.3" 2021-04-11T07:26:53.201 INFO:teuthology.orchestra.run.smithi159.stdout:PLATFORM_ID="platform:el8" 2021-04-11T07:26:53.201 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-04-11T07:26:53.202 INFO:teuthology.orchestra.run.smithi159.stdout:ANSI_COLOR="0;31" 2021-04-11T07:26:53.202 INFO:teuthology.orchestra.run.smithi159.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-04-11T07:26:53.202 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://www.redhat.com/" 2021-04-11T07:26:53.202 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-04-11T07:26:53.203 INFO:teuthology.orchestra.run.smithi159.stdout: 2021-04-11T07:26:53.203 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-04-11T07:26:53.203 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-04-11T07:26:53.203 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-04-11T07:26:53.204 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-04-11T07:26:53.205 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2021-04-11T07:26:53.226 INFO:teuthology.lock.ops:Updating [smithi201.front.sepia.ceph.com]: set os type and version on server 2021-04-11T07:26:53.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-11T07:26:53.517 INFO:teuthology.orchestra.run.smithi201.stdout:x86_64 2021-04-11T07:26:53.577 INFO:teuthology.orchestra.run.smithi201.stdout:NAME="Red Hat Enterprise Linux" 2021-04-11T07:26:53.578 INFO:teuthology.orchestra.run.smithi201.stdout:VERSION="8.3 (Ootpa)" 2021-04-11T07:26:53.578 INFO:teuthology.orchestra.run.smithi201.stdout:ID="rhel" 2021-04-11T07:26:53.578 INFO:teuthology.orchestra.run.smithi201.stdout:ID_LIKE="fedora" 2021-04-11T07:26:53.579 INFO:teuthology.orchestra.run.smithi201.stdout:VERSION_ID="8.3" 2021-04-11T07:26:53.579 INFO:teuthology.orchestra.run.smithi201.stdout:PLATFORM_ID="platform:el8" 2021-04-11T07:26:53.580 INFO:teuthology.orchestra.run.smithi201.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-04-11T07:26:53.580 INFO:teuthology.orchestra.run.smithi201.stdout:ANSI_COLOR="0;31" 2021-04-11T07:26:53.580 INFO:teuthology.orchestra.run.smithi201.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-04-11T07:26:53.581 INFO:teuthology.orchestra.run.smithi201.stdout:HOME_URL="https://www.redhat.com/" 2021-04-11T07:26:53.581 INFO:teuthology.orchestra.run.smithi201.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-04-11T07:26:53.581 INFO:teuthology.orchestra.run.smithi201.stdout: 2021-04-11T07:26:53.581 INFO:teuthology.orchestra.run.smithi201.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-04-11T07:26:53.582 INFO:teuthology.orchestra.run.smithi201.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-04-11T07:26:53.582 INFO:teuthology.orchestra.run.smithi201.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-04-11T07:26:53.582 INFO:teuthology.orchestra.run.smithi201.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-04-11T07:26:53.583 INFO:teuthology.lock.ops:Updating smithi201.front.sepia.ceph.com on lock server 2021-04-11T07:26:53.697 INFO:teuthology.dispatcher.supervisor:Running job 6034472 2021-04-11T07:26:53.698 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-04-11_04:17:02-fs-pacific-distro-basic-smithi/6034472 --name teuthology-2021-04-11_04:17:02-fs-pacific-distro-basic-smithi --description fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} -- /home/teuthworker/archive/teuthology-2021-04-11_04:17:02-fs-pacific-distro-basic-smithi/6034472/orig.config.yaml 2021-04-11T07:26:53.705 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-04-11_04:17:02-fs-pacific-distro-basic-smithi/6034472 2021-04-11T07:26:53.706 INFO:teuthology.dispatcher.supervisor:Job PID: 5537 2021-04-11T07:26:53.707 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-04-11T19:28:52.507 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-04-11T19:28:52.756 INFO:teuthology.kill:Killing Pids: {5537} 2021-04-11T19:28:53.925 INFO:teuthology.kill:Nuking machines: ['smithi159', 'smithi201'] 2021-04-11T19:28:54.885 INFO:teuthology.kill:2021-04-11 19:28:54,884.884 INFO:teuthology.nuke:targets: 2021-04-11T19:28:54.886 INFO:teuthology.kill: smithi159.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD4R4m0uCiOSIfCJeUsw1S4kkTxESu/jFf6Vz+rr4WRU9sf5QvhqYr60IlOYsCx95bNpEFTe3dRfApvJueS6MzhrFp28hYrfFrNqPN5h3PxKXLM6bQx+SZgSMkBZy+EWp8LA2UxB03yYzH1vy3yNYpwP58z2ScA18DYiUu5S+tKiOZo4/XROEoadFy3/CklredrfpeAHfurKsMZsuBDTy3NyY7ode6Bqs+BTvL3H/ij2vRYwFtc23+qLZGLCiH5jvqJyW68aknWvEAfg1AsXv3X5IxNm9E+cJutYqzo72a0B8yg3UvnRiBTlyo3uMtHJZFxD94Vf3dk7g8Yt0r6wyqjEa9tHld9NuBpE0U4dGk07QDziAB2+37OH673yxl+80ydFmAOXPjAmlTisr/MRvR1KIVoe0lnLSvmcG6Qg6LruTPpB6Gd0pVtKhYiOzaUo82DEgwkAPng99MMzXE/ka9VCoV9/ie2M9tklV7tsLhAiNLZUmCJKvyTyw9q2MP0RNE= 2021-04-11T19:28:54.886 INFO:teuthology.kill: smithi201.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC82w5HI+w+cUkWN716cs8889Zomgf9eG8nJCcRwIfnYk3SfSu2JdxdFHMOL9ApkwZbDwFh2sQTCj3SEX/iTVfYc9ziCa2ZRbd41AA7vYdfseIhFXjmOCAHceBouSiMeJr/3vnZ8oW/6MDum37j1+RY0pTG091Q+9QxkKxOrkXg3bfPkMG/jmsADwI9RrYv0363+8Lr6+WMJ44d4d6bTupGHlg6CWOcZWA/9KIrsMeY1JkbjSFvfjWt8q8DBQYy+DIA7lUovvTuVxx8wkW5KnSLBt9FyaRmQSbwcvyQjcShb+Oj2KIptiulU6m0wWsPULKC53fApuYGiPs0aqUdH3VcPxHZ2fBHWaI9w0yOW/7DrX/8sHbjkrctZ9Ua3LDZ9/M1gom9mCLEMy0MFo3WAzu+aXqIMjs9x+J5Lht/pOnHU06awbVFHTAXwHi7IGbV2GAIeGU7pLrqpvyMq3O20/KB0zeQhFvumRkM6IWsl/cUmzEsEDPhCKL09QBfY0eg3Zk= 2021-04-11T19:28:54.886 INFO:teuthology.kill:2021-04-11 19:28:54,885.885 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-11T19:28:54.887 INFO:teuthology.kill:2021-04-11 19:28:54,887.887 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-11T19:28:54.960 INFO:teuthology.kill:2021-04-11 19:28:54,960.960 INFO:teuthology.orchestra.console:Power off smithi159 2021-04-11T19:28:54.973 INFO:teuthology.kill:2021-04-11 19:28:54,973.973 INFO:teuthology.orchestra.console:Power off smithi201 2021-04-11T19:29:07.443 INFO:teuthology.kill:2021-04-11 19:29:07,442.442 INFO:teuthology.orchestra.console:Power off for smithi159 completed 2021-04-11T19:29:07.450 INFO:teuthology.kill:2021-04-11 19:29:07,450.450 INFO:teuthology.orchestra.console:Power off for smithi201 completed 2021-04-11T19:29:07.687 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-04-11T19:29:07.688 INFO:teuthology.task.internal:roles: ubuntu@smithi201.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2021-04-11T19:29:07.688 INFO:teuthology.misc:Compressing logs... 2021-04-11T19:29:07.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-11T19:29:26.108 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-04-11T19:29:26.193 INFO:teuthology.kill:No teuthology processes running 2021-04-11T19:29:27.465 INFO:teuthology.kill:Nuking machines: ['smithi159', 'smithi201'] 2021-04-11T19:29:28.387 INFO:teuthology.kill:2021-04-11 19:29:28,387.387 INFO:teuthology.nuke:targets: 2021-04-11T19:29:28.388 INFO:teuthology.kill: smithi159.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD4R4m0uCiOSIfCJeUsw1S4kkTxESu/jFf6Vz+rr4WRU9sf5QvhqYr60IlOYsCx95bNpEFTe3dRfApvJueS6MzhrFp28hYrfFrNqPN5h3PxKXLM6bQx+SZgSMkBZy+EWp8LA2UxB03yYzH1vy3yNYpwP58z2ScA18DYiUu5S+tKiOZo4/XROEoadFy3/CklredrfpeAHfurKsMZsuBDTy3NyY7ode6Bqs+BTvL3H/ij2vRYwFtc23+qLZGLCiH5jvqJyW68aknWvEAfg1AsXv3X5IxNm9E+cJutYqzo72a0B8yg3UvnRiBTlyo3uMtHJZFxD94Vf3dk7g8Yt0r6wyqjEa9tHld9NuBpE0U4dGk07QDziAB2+37OH673yxl+80ydFmAOXPjAmlTisr/MRvR1KIVoe0lnLSvmcG6Qg6LruTPpB6Gd0pVtKhYiOzaUo82DEgwkAPng99MMzXE/ka9VCoV9/ie2M9tklV7tsLhAiNLZUmCJKvyTyw9q2MP0RNE= 2021-04-11T19:29:28.388 INFO:teuthology.kill: smithi201.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC82w5HI+w+cUkWN716cs8889Zomgf9eG8nJCcRwIfnYk3SfSu2JdxdFHMOL9ApkwZbDwFh2sQTCj3SEX/iTVfYc9ziCa2ZRbd41AA7vYdfseIhFXjmOCAHceBouSiMeJr/3vnZ8oW/6MDum37j1+RY0pTG091Q+9QxkKxOrkXg3bfPkMG/jmsADwI9RrYv0363+8Lr6+WMJ44d4d6bTupGHlg6CWOcZWA/9KIrsMeY1JkbjSFvfjWt8q8DBQYy+DIA7lUovvTuVxx8wkW5KnSLBt9FyaRmQSbwcvyQjcShb+Oj2KIptiulU6m0wWsPULKC53fApuYGiPs0aqUdH3VcPxHZ2fBHWaI9w0yOW/7DrX/8sHbjkrctZ9Ua3LDZ9/M1gom9mCLEMy0MFo3WAzu+aXqIMjs9x+J5Lht/pOnHU06awbVFHTAXwHi7IGbV2GAIeGU7pLrqpvyMq3O20/KB0zeQhFvumRkM6IWsl/cUmzEsEDPhCKL09QBfY0eg3Zk= 2021-04-11T19:29:28.409 INFO:teuthology.kill:2021-04-11 19:29:28,408.408 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-11T19:29:28.422 INFO:teuthology.kill:2021-04-11 19:29:28,421.421 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-11T19:29:28.484 INFO:teuthology.kill:2021-04-11 19:29:28,483.483 INFO:teuthology.orchestra.console:Power off smithi159 2021-04-11T19:29:28.497 INFO:teuthology.kill:2021-04-11 19:29:28,496.496 INFO:teuthology.orchestra.console:Power off smithi201 2021-04-11T19:29:28.631 INFO:teuthology.kill:2021-04-11 19:29:28,631.631 INFO:teuthology.orchestra.console:Power off for smithi159 completed 2021-04-11T19:29:28.642 INFO:teuthology.kill:2021-04-11 19:29:28,642.642 INFO:teuthology.orchestra.console:Power off for smithi201 completed 2021-04-11T19:29:28.781 INFO:teuthology.kill:2021-04-11 19:29:28,780.780 INFO:teuthology.lock.ops:unlocked smithi159.front.sepia.ceph.com 2021-04-11T19:29:28.797 INFO:teuthology.kill:2021-04-11 19:29:28,797.797 INFO:teuthology.lock.ops:unlocked smithi201.front.sepia.ceph.com 2021-04-11T19:31:29.132 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-04-11T19:31:29.249 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-04-11T19:31:29.249 INFO:teuthology.nuke:Checking targets against current locks 2021-04-11T19:31:29.325 INFO:teuthology.nuke:Not nuking smithi201.front.sepia.ceph.com because description doesn't match 2021-04-11T19:31:29.326 INFO:teuthology.nuke:Not nuking smithi159.front.sepia.ceph.com because description doesn't match