2021-04-11T08:12:43.317 INFO:root:teuthology version: 1.1.0-85d61eae 2021-04-11T08:12:43.528 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi065.front.sepia.ceph.com' reimaging 2021-04-11T08:12:43.529 INFO:teuthology.lock.ops:Updating [ubuntu@smithi065.front.sepia.ceph.com]: reset os type and version on server 2021-04-11T08:12:43.530 INFO:teuthology.lock.ops:Updating smithi065.front.sepia.ceph.com on lock server 2021-04-11T08:12:43.563 INFO:teuthology.lock.ops:Node 'ubuntu@smithi065.front.sepia.ceph.com' reimaging is complete 2021-04-11T08:12:43.564 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi160.front.sepia.ceph.com' reimaging 2021-04-11T08:12:43.564 INFO:teuthology.lock.ops:Updating [ubuntu@smithi160.front.sepia.ceph.com]: reset os type and version on server 2021-04-11T08:12:43.564 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2021-04-11T08:12:43.588 INFO:teuthology.lock.ops:Node 'ubuntu@smithi160.front.sepia.ceph.com' reimaging is complete 2021-04-11T08:12:43.871 INFO:teuthology.provision.fog.smithi065:Scheduling deploy of rhel 8.3 2021-04-11T08:12:43.891 INFO:teuthology.provision.fog.smithi160:Scheduling deploy of rhel 8.3 2021-04-11T08:12:44.771 INFO:teuthology.orchestra.console:Power off smithi160 2021-04-11T08:12:44.929 INFO:teuthology.orchestra.console:Power off smithi065 2021-04-11T08:12:45.004 INFO:teuthology.orchestra.console:Power off for smithi160 completed 2021-04-11T08:12:45.106 INFO:teuthology.orchestra.console:Power on smithi160 2021-04-11T08:12:45.127 INFO:teuthology.orchestra.console:Power off for smithi065 completed 2021-04-11T08:12:45.229 INFO:teuthology.orchestra.console:Power on smithi065 2021-04-11T08:12:49.520 INFO:teuthology.orchestra.console:Power on for smithi065 completed 2021-04-11T08:12:49.623 INFO:teuthology.provision.fog.smithi065:Waiting for deploy to finish 2021-04-11T08:12:53.534 INFO:teuthology.orchestra.console:Power on for smithi160 completed 2021-04-11T08:12:53.636 INFO:teuthology.provision.fog.smithi160:Waiting for deploy to finish 2021-04-11T08:17:04.821 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-11T08:17:25.803 INFO:teuthology.provision.fog.smithi065:Node is ready 2021-04-11T08:17:25.824 INFO:teuthology.orchestra.run.smithi065.stdout:smithi065.front.sepia.ceph.com 2021-04-11T08:17:25.881 INFO:teuthology.orchestra.run.smithi065.stdout:172.21.15.65 smithi065.front.sepia.ceph.com smithi065 2021-04-11T08:17:26.234 INFO:teuthology.provision.fog.smithi065:Deploy complete! 2021-04-11T08:17:41.049 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-11T08:17:41.337 INFO:teuthology.provision.fog.smithi160:Node is ready 2021-04-11T08:17:41.355 INFO:teuthology.orchestra.run.smithi160.stdout:smithi160.front.sepia.ceph.com 2021-04-11T08:17:41.415 INFO:teuthology.orchestra.run.smithi160.stdout:172.21.15.160 smithi160.front.sepia.ceph.com smithi160 2021-04-11T08:17:41.742 INFO:teuthology.provision.fog.smithi160:Deploy complete! 2021-04-11T08:17:41.885 INFO:teuthology.lock.ops:Checking smithi065.front.sepia.ceph.com 2021-04-11T08:17:41.898 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-11T08:17:41.920 INFO:teuthology.lock.ops:Checking smithi160.front.sepia.ceph.com 2021-04-11T08:17:41.921 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-11T08:17:41.936 INFO:teuthology.lock.ops:Updating [smithi065.front.sepia.ceph.com]: set os type and version on server 2021-04-11T08:17:41.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-11T08:17:42.234 INFO:teuthology.orchestra.run.smithi065.stdout:x86_64 2021-04-11T08:17:42.289 INFO:teuthology.orchestra.run.smithi065.stdout:NAME="Red Hat Enterprise Linux" 2021-04-11T08:17:42.289 INFO:teuthology.orchestra.run.smithi065.stdout:VERSION="8.3 (Ootpa)" 2021-04-11T08:17:42.290 INFO:teuthology.orchestra.run.smithi065.stdout:ID="rhel" 2021-04-11T08:17:42.290 INFO:teuthology.orchestra.run.smithi065.stdout:ID_LIKE="fedora" 2021-04-11T08:17:42.290 INFO:teuthology.orchestra.run.smithi065.stdout:VERSION_ID="8.3" 2021-04-11T08:17:42.290 INFO:teuthology.orchestra.run.smithi065.stdout:PLATFORM_ID="platform:el8" 2021-04-11T08:17:42.291 INFO:teuthology.orchestra.run.smithi065.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-04-11T08:17:42.291 INFO:teuthology.orchestra.run.smithi065.stdout:ANSI_COLOR="0;31" 2021-04-11T08:17:42.291 INFO:teuthology.orchestra.run.smithi065.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-04-11T08:17:42.292 INFO:teuthology.orchestra.run.smithi065.stdout:HOME_URL="https://www.redhat.com/" 2021-04-11T08:17:42.292 INFO:teuthology.orchestra.run.smithi065.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-04-11T08:17:42.292 INFO:teuthology.orchestra.run.smithi065.stdout: 2021-04-11T08:17:42.292 INFO:teuthology.orchestra.run.smithi065.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-04-11T08:17:42.292 INFO:teuthology.orchestra.run.smithi065.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-04-11T08:17:42.293 INFO:teuthology.orchestra.run.smithi065.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-04-11T08:17:42.293 INFO:teuthology.orchestra.run.smithi065.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-04-11T08:17:42.294 INFO:teuthology.lock.ops:Updating smithi065.front.sepia.ceph.com on lock server 2021-04-11T08:17:42.310 INFO:teuthology.lock.ops:Updating [smithi160.front.sepia.ceph.com]: set os type and version on server 2021-04-11T08:17:42.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-11T08:17:42.587 INFO:teuthology.orchestra.run.smithi160.stdout:x86_64 2021-04-11T08:17:42.641 INFO:teuthology.orchestra.run.smithi160.stdout:NAME="Red Hat Enterprise Linux" 2021-04-11T08:17:42.641 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION="8.3 (Ootpa)" 2021-04-11T08:17:42.642 INFO:teuthology.orchestra.run.smithi160.stdout:ID="rhel" 2021-04-11T08:17:42.642 INFO:teuthology.orchestra.run.smithi160.stdout:ID_LIKE="fedora" 2021-04-11T08:17:42.642 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_ID="8.3" 2021-04-11T08:17:42.642 INFO:teuthology.orchestra.run.smithi160.stdout:PLATFORM_ID="platform:el8" 2021-04-11T08:17:42.643 INFO:teuthology.orchestra.run.smithi160.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-04-11T08:17:42.643 INFO:teuthology.orchestra.run.smithi160.stdout:ANSI_COLOR="0;31" 2021-04-11T08:17:42.643 INFO:teuthology.orchestra.run.smithi160.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-04-11T08:17:42.643 INFO:teuthology.orchestra.run.smithi160.stdout:HOME_URL="https://www.redhat.com/" 2021-04-11T08:17:42.644 INFO:teuthology.orchestra.run.smithi160.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-04-11T08:17:42.644 INFO:teuthology.orchestra.run.smithi160.stdout: 2021-04-11T08:17:42.644 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-04-11T08:17:42.644 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-04-11T08:17:42.645 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-04-11T08:17:42.645 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-04-11T08:17:42.646 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2021-04-11T08:17:42.748 INFO:teuthology.dispatcher.supervisor:Running job 6034549 2021-04-11T08:17:42.748 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-04-11_04:17:02-fs-pacific-distro-basic-smithi/6034549 --name teuthology-2021-04-11_04:17:02-fs-pacific-distro-basic-smithi --description fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} -- /home/teuthworker/archive/teuthology-2021-04-11_04:17:02-fs-pacific-distro-basic-smithi/6034549/orig.config.yaml 2021-04-11T08:17:42.752 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-04-11_04:17:02-fs-pacific-distro-basic-smithi/6034549 2021-04-11T08:17:42.753 INFO:teuthology.dispatcher.supervisor:Job PID: 14476 2021-04-11T08:17:42.754 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-04-11T20:19:28.152 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-04-11T20:19:28.425 INFO:teuthology.kill:Killing Pids: {14476} 2021-04-11T20:19:29.628 INFO:teuthology.kill:Nuking machines: ['smithi065', 'smithi160'] 2021-04-11T20:19:30.600 INFO:teuthology.kill:2021-04-11 20:19:30,599.599 INFO:teuthology.nuke:targets: 2021-04-11T20:19:30.601 INFO:teuthology.kill: smithi065.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8alIyjHENkFtplzsaqB+MfC0MKnmLBNEY4hc6c15a2WlNIz4+tTJROExIGmfTMPzTLvM+Uz6aM+q93P2W2wH9971hCkgmYW5OF6v3XmhIBUFR4/F6E2WRTW5MdvN4BcIB+BnOb74JmEkWJmV0nG4l+tMfU0ScdNQxpXNHlOieb1lX8lRqN7HOVb4jgP5YbGq/2gH+vFfjXXXyO22eSJCYUVgmVsauGSBbIWw2AmMKh43jXWvzN8uDWs+KgNBrXHUbfqpCtQq3NH2rH/cbRN05AXbHuQmPGZopnYTddPqTaSSLGiUYKmn2KdSatJk8ShCl3s6vQkohECJ2ff/7xZ14BxIiURAvBCp6yadcBPwRf3EVx6q9OIim3UR5wMNyF41TLJuDd7/oNV9g5wU7lDE7AK0Hyr+EUDGJzZNy7gVglMczF8TTknqDqtmDjjg1t13OIYKCYdqM5x1qXXMZUIUjHWpx47w5IZpyMt5wVAcHWsL3+Ni+dMGfjInML9XPAMc= 2021-04-11T20:19:30.602 INFO:teuthology.kill: smithi160.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrRbGe8xrPls0XptQIm49d6Xu1wfWOKzbca/xXMGHrUSm9H8gGXW5TF8Fsk8d2vpyr/kT70MXLsqU7z+iCmI1mfgWhNxpC3w1yIU8nce0559HaKg1YPc5ejTwv76zG3WESCqToi5kTkZ2mEEvEfDy8CGwPfTwxTEhkB8PCoC2gKvdajDNvAj+gV47KdC68jGbj//VXNSohfk2wY0JANG1uOJf5JWg0D9sSYsC/Cpd9+p3uMtw2+F+Bx2IWkAQOJEYVbrs4IpkbmZtRf8oeN3GNplyWqKsHxd0napgQTifEaTNgs0ot+TvosAXX6Ir4QmHK1COOb7P8dKhPXRoUR04OgLeSZ4D2bGxlfaISsumQ5oyImh8cLOsMUBzvZp07zzHH7RaIx4X0uaeON3uCkfSORN2ZlmcGXxM+oDE158OrGp3ZPSaL2eahcESmC7ZeoH5Bj/zYRRny3wrFTw9uTuebMQfICWru/UgjXP45DJkadz6q1odpJZt0BQkpmkb2j9U= 2021-04-11T20:19:30.602 INFO:teuthology.kill:2021-04-11 20:19:30,600.600 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-11T20:19:30.603 INFO:teuthology.kill:2021-04-11 20:19:30,602.602 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-11T20:19:30.681 INFO:teuthology.kill:2021-04-11 20:19:30,680.680 INFO:teuthology.orchestra.console:Power off smithi065 2021-04-11T20:19:30.702 INFO:teuthology.kill:2021-04-11 20:19:30,702.702 INFO:teuthology.orchestra.console:Power off smithi160 2021-04-11T20:19:39.068 INFO:teuthology.kill:2021-04-11 20:19:39,067.067 INFO:teuthology.orchestra.console:Power off for smithi160 completed 2021-04-11T20:19:43.172 INFO:teuthology.kill:2021-04-11 20:19:43,172.172 INFO:teuthology.orchestra.console:Power off for smithi065 completed 2021-04-11T20:19:43.465 INFO:teuthology.task.internal:roles: ubuntu@smithi065.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-04-11T20:19:43.465 INFO:teuthology.task.internal:roles: ubuntu@smithi160.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2021-04-11T20:19:43.466 INFO:teuthology.misc:Compressing logs... 2021-04-11T20:19:43.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-11T20:20:43.528 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-04-11T20:20:43.615 INFO:teuthology.kill:No teuthology processes running 2021-04-11T20:20:44.907 INFO:teuthology.kill:Nuking machines: ['smithi065', 'smithi160'] 2021-04-11T20:20:45.954 INFO:teuthology.kill:2021-04-11 20:20:45,954.954 INFO:teuthology.nuke:targets: 2021-04-11T20:20:45.955 INFO:teuthology.kill: smithi065.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8alIyjHENkFtplzsaqB+MfC0MKnmLBNEY4hc6c15a2WlNIz4+tTJROExIGmfTMPzTLvM+Uz6aM+q93P2W2wH9971hCkgmYW5OF6v3XmhIBUFR4/F6E2WRTW5MdvN4BcIB+BnOb74JmEkWJmV0nG4l+tMfU0ScdNQxpXNHlOieb1lX8lRqN7HOVb4jgP5YbGq/2gH+vFfjXXXyO22eSJCYUVgmVsauGSBbIWw2AmMKh43jXWvzN8uDWs+KgNBrXHUbfqpCtQq3NH2rH/cbRN05AXbHuQmPGZopnYTddPqTaSSLGiUYKmn2KdSatJk8ShCl3s6vQkohECJ2ff/7xZ14BxIiURAvBCp6yadcBPwRf3EVx6q9OIim3UR5wMNyF41TLJuDd7/oNV9g5wU7lDE7AK0Hyr+EUDGJzZNy7gVglMczF8TTknqDqtmDjjg1t13OIYKCYdqM5x1qXXMZUIUjHWpx47w5IZpyMt5wVAcHWsL3+Ni+dMGfjInML9XPAMc= 2021-04-11T20:20:45.955 INFO:teuthology.kill: smithi160.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrRbGe8xrPls0XptQIm49d6Xu1wfWOKzbca/xXMGHrUSm9H8gGXW5TF8Fsk8d2vpyr/kT70MXLsqU7z+iCmI1mfgWhNxpC3w1yIU8nce0559HaKg1YPc5ejTwv76zG3WESCqToi5kTkZ2mEEvEfDy8CGwPfTwxTEhkB8PCoC2gKvdajDNvAj+gV47KdC68jGbj//VXNSohfk2wY0JANG1uOJf5JWg0D9sSYsC/Cpd9+p3uMtw2+F+Bx2IWkAQOJEYVbrs4IpkbmZtRf8oeN3GNplyWqKsHxd0napgQTifEaTNgs0ot+TvosAXX6Ir4QmHK1COOb7P8dKhPXRoUR04OgLeSZ4D2bGxlfaISsumQ5oyImh8cLOsMUBzvZp07zzHH7RaIx4X0uaeON3uCkfSORN2ZlmcGXxM+oDE158OrGp3ZPSaL2eahcESmC7ZeoH5Bj/zYRRny3wrFTw9uTuebMQfICWru/UgjXP45DJkadz6q1odpJZt0BQkpmkb2j9U= 2021-04-11T20:20:46.109 INFO:teuthology.kill:2021-04-11 20:20:46,109.109 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-11T20:20:46.121 INFO:teuthology.kill:2021-04-11 20:20:46,121.121 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-11T20:20:46.221 INFO:teuthology.kill:2021-04-11 20:20:46,221.221 INFO:teuthology.orchestra.console:Power off smithi065 2021-04-11T20:20:46.234 INFO:teuthology.kill:2021-04-11 20:20:46,234.234 INFO:teuthology.orchestra.console:Power off smithi160 2021-04-11T20:20:46.370 INFO:teuthology.kill:2021-04-11 20:20:46,369.369 INFO:teuthology.orchestra.console:Power off for smithi065 completed 2021-04-11T20:20:46.382 INFO:teuthology.kill:2021-04-11 20:20:46,382.382 INFO:teuthology.orchestra.console:Power off for smithi160 completed 2021-04-11T20:20:46.520 INFO:teuthology.kill:2021-04-11 20:20:46,520.520 INFO:teuthology.lock.ops:unlocked smithi065.front.sepia.ceph.com 2021-04-11T20:20:46.539 INFO:teuthology.kill:2021-04-11 20:20:46,539.539 INFO:teuthology.lock.ops:unlocked smithi160.front.sepia.ceph.com 2021-04-11T20:22:46.833 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-04-11T20:22:46.951 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-04-11T20:22:46.952 INFO:teuthology.nuke:Checking targets against current locks 2021-04-11T20:22:47.032 INFO:teuthology.nuke:Not nuking smithi065.front.sepia.ceph.com because description doesn't match 2021-04-11T20:22:47.033 INFO:teuthology.nuke:Not nuking smithi160.front.sepia.ceph.com because description doesn't match