2021-05-23T01:48:46.559 INFO:root:teuthology version: 1.1.0-d92ec140 2021-05-23T01:48:46.657 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi067.front.sepia.ceph.com' reimaging 2021-05-23T01:48:46.658 INFO:teuthology.lock.ops:Updating [ubuntu@smithi067.front.sepia.ceph.com]: reset os type and version on server 2021-05-23T01:48:46.659 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2021-05-23T01:48:46.678 INFO:teuthology.lock.ops:Node 'ubuntu@smithi067.front.sepia.ceph.com' reimaging is complete 2021-05-23T01:48:46.678 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi187.front.sepia.ceph.com' reimaging 2021-05-23T01:48:46.679 INFO:teuthology.lock.ops:Updating [ubuntu@smithi187.front.sepia.ceph.com]: reset os type and version on server 2021-05-23T01:48:46.679 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2021-05-23T01:48:46.698 INFO:teuthology.lock.ops:Node 'ubuntu@smithi187.front.sepia.ceph.com' reimaging is complete 2021-05-23T01:48:46.874 INFO:teuthology.provision.fog.smithi067:Scheduling deploy of centos 8.3 2021-05-23T01:48:46.896 INFO:teuthology.provision.fog.smithi187:Scheduling deploy of centos 8.3 2021-05-23T01:48:47.227 INFO:teuthology.orchestra.console:Power off smithi067 2021-05-23T01:48:47.311 INFO:teuthology.orchestra.console:Power off smithi187 2021-05-23T01:48:47.371 INFO:teuthology.orchestra.console:Power off for smithi067 completed 2021-05-23T01:48:47.453 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2021-05-23T01:48:47.472 INFO:teuthology.orchestra.console:Power on smithi067 2021-05-23T01:48:47.554 INFO:teuthology.orchestra.console:Power on smithi187 2021-05-23T01:48:51.807 INFO:teuthology.orchestra.console:Power on for smithi187 completed 2021-05-23T01:48:51.908 INFO:teuthology.provision.fog.smithi187:Waiting for deploy to finish 2021-05-23T01:48:55.837 INFO:teuthology.orchestra.console:Power on for smithi067 completed 2021-05-23T01:48:55.938 INFO:teuthology.provision.fog.smithi067:Waiting for deploy to finish 2021-05-23T01:53:23.835 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-23T01:53:24.158 INFO:teuthology.provision.fog.smithi187:Node is ready 2021-05-23T01:53:24.213 INFO:teuthology.orchestra.run.smithi187.stdout:smithi187.front.sepia.ceph.com 2021-05-23T01:53:24.268 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.15.187 smithi187.front.sepia.ceph.com smithi187 2021-05-23T01:53:24.585 INFO:teuthology.provision.fog.smithi187:Deploy complete! 2021-05-23T01:53:36.741 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-23T01:53:37.050 INFO:teuthology.provision.fog.smithi067:Node is ready 2021-05-23T01:53:37.069 INFO:teuthology.orchestra.run.smithi067.stdout:smithi067.front.sepia.ceph.com 2021-05-23T01:53:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67 smithi067.front.sepia.ceph.com smithi067 2021-05-23T01:53:37.466 INFO:teuthology.provision.fog.smithi067:Deploy complete! 2021-05-23T01:53:37.585 INFO:teuthology.lock.ops:Checking smithi067.front.sepia.ceph.com 2021-05-23T01:53:37.587 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-23T01:53:37.605 INFO:teuthology.lock.ops:Checking smithi187.front.sepia.ceph.com 2021-05-23T01:53:37.606 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-23T01:53:37.627 INFO:teuthology.lock.ops:Updating [smithi067.front.sepia.ceph.com]: set os type and version on server 2021-05-23T01:53:37.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-23T01:53:37.913 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2021-05-23T01:53:37.968 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Linux" 2021-05-23T01:53:37.968 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="8" 2021-05-23T01:53:37.968 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2021-05-23T01:53:37.969 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2021-05-23T01:53:37.969 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="8" 2021-05-23T01:53:37.969 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el8" 2021-05-23T01:53:37.969 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Linux 8" 2021-05-23T01:53:37.970 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2021-05-23T01:53:37.970 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-23T01:53:37.970 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2021-05-23T01:53:37.970 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-23T01:53:37.971 INFO:teuthology.orchestra.run.smithi067.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-23T01:53:37.971 INFO:teuthology.orchestra.run.smithi067.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-23T01:53:37.972 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2021-05-23T01:53:37.989 INFO:teuthology.lock.ops:Updating [smithi187.front.sepia.ceph.com]: set os type and version on server 2021-05-23T01:53:37.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-23T01:53:38.276 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2021-05-23T01:53:38.331 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Linux" 2021-05-23T01:53:38.331 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="8" 2021-05-23T01:53:38.332 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos" 2021-05-23T01:53:38.332 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora" 2021-05-23T01:53:38.332 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="8" 2021-05-23T01:53:38.332 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el8" 2021-05-23T01:53:38.333 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Linux 8" 2021-05-23T01:53:38.333 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2021-05-23T01:53:38.333 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-23T01:53:38.334 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://centos.org/" 2021-05-23T01:53:38.335 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-23T01:53:38.335 INFO:teuthology.orchestra.run.smithi187.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-23T01:53:38.335 INFO:teuthology.orchestra.run.smithi187.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-23T01:53:38.336 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2021-05-23T01:53:38.415 INFO:teuthology.dispatcher.supervisor:Running job 6129219 2021-05-23T01:53:38.416 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/virtualenv/bin/teuthology -v --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2021-05-23_00:57:16-fs-wip-kefu-testing-2021-05-22-2207-distro-basic-smithi/6129219 --name pdonnell-2021-05-23_00:57:16-fs-wip-kefu-testing-2021-05-22-2207-distro-basic-smithi --description fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-full} -- /home/teuthworker/archive/pdonnell-2021-05-23_00:57:16-fs-wip-kefu-testing-2021-05-22-2207-distro-basic-smithi/6129219/orig.config.yaml 2021-05-23T01:53:38.420 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/pdonnell-2021-05-23_00:57:16-fs-wip-kefu-testing-2021-05-22-2207-distro-basic-smithi/6129219 2021-05-23T01:53:38.421 INFO:teuthology.dispatcher.supervisor:Job PID: 10634 2021-05-23T01:53:38.421 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-23T13:54:26.231 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-23T13:54:26.485 INFO:teuthology.kill:Killing Pids: {10634} 2021-05-23T13:54:27.556 INFO:teuthology.kill:Nuking machines: ['smithi067', 'smithi187'] 2021-05-23T13:54:28.447 INFO:teuthology.kill:2021-05-23 13:54:28,446.446 INFO:teuthology.nuke:targets: 2021-05-23T13:54:28.448 INFO:teuthology.kill: smithi067.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmWViIU/KFF/9ayzAIYBQOsRNpXdx7iVypdMIADcXkb4/N06HNzULG/4EPv6/axcsSwrkJpTHvn+Gy9yaGCPF5qGWWbdIeODA7x/ojcT2Pzt5gGGqolEWEGYYasM3oQ9LTr/5mFYvGMonicbttKGDUwNIcPzz45JjtgR+kKHHPt8Zu/3EH+1R5xzveLFUFLZpa7ZP5AleCp62mF0/mI4lAX0du3vFuEpSqpCvJt/6CIUuRS9qxAys7omXFW45QhcfUyod8DL0KFwJM2zWYmGxzgHgRAkBk1DGZjckQTLxk/K/OGPSvnQ+1JotbmMvQ5Md4Xq81WYMwlOQMki3yvhNIYCkW8WfRWFHWKrkRUGoxrHqXKd/WlzPaaX0sdMKWuPuKckHGa1vnmbwFNXVR4H8qoLMW8hVJLCIFVYFBXCX2ZW4HM6iY9909MgOucuS8tqQV2rdPChtasw55apJiBA2o/gPqpXOQXpg2YlOnrwDgFnEq7/6eHRye1Q+QKZWEDBE= 2021-05-23T13:54:28.448 INFO:teuthology.kill: smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDn2/aGEMLls4AqFBwjZ2LR931t5rl1FUzqQxxX9oihO/HxnuXAs/5PbsPV3MQfLbi1VIKIZNfGRvCpjO95v54aVJxdnke/cT936phLPBYEeF5gLhFyD4wxIh40cLyZHeV97+4HtV2cCYqBYiBeoKAoyLHlK6f5Lcs9I/Kld/MdLh9fYNqRWB355B4p7fcn5tsivhgA06LmGt09bykKb2yUuQVTLpUNMXorr3QKdxSUqElposR+3G1gWupsqgYY7fzHfIMt4+ZZjtsFwqx3IOIR+yVn/1I7AZVqBdkO5WUwNA28ZPiT2vESoHWVDAOj0QL10baZOCvNoHl+qKtPP65gH0eqQkh7/n53ki+79imyZxI5o30BnasU6K8lwLo0jOe6dnd3pb6xPbXaaxKBPMHFzbtpg1i5Enl2VH7EXOgwBwkqahgNhrSa0CuEQrKFNwWAwmBA60WxJdMhsIPn6mkDG07ZFsLUSw11yFo7p08fbroGmDfJk1QC3GfQr6ExBBc= 2021-05-23T13:54:28.449 INFO:teuthology.kill:2021-05-23 13:54:28,447.447 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-23T13:54:28.449 INFO:teuthology.kill:2021-05-23 13:54:28,449.449 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-23T13:54:28.507 INFO:teuthology.kill:2021-05-23 13:54:28,507.507 INFO:teuthology.orchestra.console:Power off smithi067 2021-05-23T13:54:28.521 INFO:teuthology.kill:2021-05-23 13:54:28,521.521 INFO:teuthology.orchestra.console:Power off smithi187 2021-05-23T13:54:36.885 INFO:teuthology.kill:2021-05-23 13:54:36,885.885 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2021-05-23T13:54:40.992 INFO:teuthology.kill:2021-05-23 13:54:40,991.991 INFO:teuthology.orchestra.console:Power off for smithi067 completed 2021-05-23T13:54:41.214 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2021-05-23T13:54:41.214 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2021-05-23T13:54:41.215 INFO:teuthology.misc:Compressing logs... 2021-05-23T13:54:41.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-23T13:55:41.275 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-23T13:55:41.344 INFO:teuthology.kill:No teuthology processes running 2021-05-23T13:55:42.373 INFO:teuthology.kill:Nuking machines: ['smithi067', 'smithi187'] 2021-05-23T13:55:43.272 INFO:teuthology.kill:2021-05-23 13:55:43,272.272 INFO:teuthology.nuke:targets: 2021-05-23T13:55:43.273 INFO:teuthology.kill: smithi067.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmWViIU/KFF/9ayzAIYBQOsRNpXdx7iVypdMIADcXkb4/N06HNzULG/4EPv6/axcsSwrkJpTHvn+Gy9yaGCPF5qGWWbdIeODA7x/ojcT2Pzt5gGGqolEWEGYYasM3oQ9LTr/5mFYvGMonicbttKGDUwNIcPzz45JjtgR+kKHHPt8Zu/3EH+1R5xzveLFUFLZpa7ZP5AleCp62mF0/mI4lAX0du3vFuEpSqpCvJt/6CIUuRS9qxAys7omXFW45QhcfUyod8DL0KFwJM2zWYmGxzgHgRAkBk1DGZjckQTLxk/K/OGPSvnQ+1JotbmMvQ5Md4Xq81WYMwlOQMki3yvhNIYCkW8WfRWFHWKrkRUGoxrHqXKd/WlzPaaX0sdMKWuPuKckHGa1vnmbwFNXVR4H8qoLMW8hVJLCIFVYFBXCX2ZW4HM6iY9909MgOucuS8tqQV2rdPChtasw55apJiBA2o/gPqpXOQXpg2YlOnrwDgFnEq7/6eHRye1Q+QKZWEDBE= 2021-05-23T13:55:43.274 INFO:teuthology.kill: smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDn2/aGEMLls4AqFBwjZ2LR931t5rl1FUzqQxxX9oihO/HxnuXAs/5PbsPV3MQfLbi1VIKIZNfGRvCpjO95v54aVJxdnke/cT936phLPBYEeF5gLhFyD4wxIh40cLyZHeV97+4HtV2cCYqBYiBeoKAoyLHlK6f5Lcs9I/Kld/MdLh9fYNqRWB355B4p7fcn5tsivhgA06LmGt09bykKb2yUuQVTLpUNMXorr3QKdxSUqElposR+3G1gWupsqgYY7fzHfIMt4+ZZjtsFwqx3IOIR+yVn/1I7AZVqBdkO5WUwNA28ZPiT2vESoHWVDAOj0QL10baZOCvNoHl+qKtPP65gH0eqQkh7/n53ki+79imyZxI5o30BnasU6K8lwLo0jOe6dnd3pb6xPbXaaxKBPMHFzbtpg1i5Enl2VH7EXOgwBwkqahgNhrSa0CuEQrKFNwWAwmBA60WxJdMhsIPn6mkDG07ZFsLUSw11yFo7p08fbroGmDfJk1QC3GfQr6ExBBc= 2021-05-23T13:55:43.292 INFO:teuthology.kill:2021-05-23 13:55:43,291.291 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-23T13:55:43.294 INFO:teuthology.kill:2021-05-23 13:55:43,294.294 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-23T13:55:43.345 INFO:teuthology.kill:2021-05-23 13:55:43,345.345 INFO:teuthology.orchestra.console:Power off smithi187 2021-05-23T13:55:43.358 INFO:teuthology.kill:2021-05-23 13:55:43,358.358 INFO:teuthology.orchestra.console:Power off smithi067 2021-05-23T13:55:43.486 INFO:teuthology.kill:2021-05-23 13:55:43,486.486 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2021-05-23T13:55:43.497 INFO:teuthology.kill:2021-05-23 13:55:43,497.497 INFO:teuthology.orchestra.console:Power off for smithi067 completed 2021-05-23T13:55:43.630 INFO:teuthology.kill:2021-05-23 13:55:43,629.629 INFO:teuthology.lock.ops:unlocked smithi187.front.sepia.ceph.com 2021-05-23T13:55:43.633 INFO:teuthology.kill:2021-05-23 13:55:43,632.632 INFO:teuthology.lock.ops:unlocked smithi067.front.sepia.ceph.com 2021-05-23T13:57:43.902 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15