2021-05-15T10:02:53.493 INFO:root:teuthology version: 1.1.0-19220a3b 2021-05-15T10:02:54.616 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi033.front.sepia.ceph.com' reimaging 2021-05-15T10:02:54.617 INFO:teuthology.lock.ops:Updating [ubuntu@smithi033.front.sepia.ceph.com]: reset os type and version on server 2021-05-15T10:02:54.618 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2021-05-15T10:02:54.639 INFO:teuthology.lock.ops:Node 'ubuntu@smithi033.front.sepia.ceph.com' reimaging is complete 2021-05-15T10:02:54.639 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi041.front.sepia.ceph.com' reimaging 2021-05-15T10:02:54.640 INFO:teuthology.lock.ops:Updating [ubuntu@smithi041.front.sepia.ceph.com]: reset os type and version on server 2021-05-15T10:02:54.640 INFO:teuthology.lock.ops:Updating smithi041.front.sepia.ceph.com on lock server 2021-05-15T10:02:54.658 INFO:teuthology.lock.ops:Node 'ubuntu@smithi041.front.sepia.ceph.com' reimaging is complete 2021-05-15T10:02:55.450 INFO:teuthology.provision.fog.smithi041:Scheduling deploy of centos 8.2 2021-05-15T10:02:55.537 INFO:teuthology.provision.fog.smithi033:Scheduling deploy of centos 8.2 2021-05-15T10:02:58.753 INFO:teuthology.orchestra.console:Power off smithi041 2021-05-15T10:02:58.827 INFO:teuthology.orchestra.console:Power off smithi033 2021-05-15T10:02:58.893 INFO:teuthology.orchestra.console:Power off for smithi041 completed 2021-05-15T10:02:58.968 INFO:teuthology.orchestra.console:Power off for smithi033 completed 2021-05-15T10:02:58.995 INFO:teuthology.orchestra.console:Power on smithi041 2021-05-15T10:02:59.070 INFO:teuthology.orchestra.console:Power on smithi033 2021-05-15T10:03:07.355 INFO:teuthology.orchestra.console:Power on for smithi041 completed 2021-05-15T10:03:07.425 INFO:teuthology.orchestra.console:Power on for smithi033 completed 2021-05-15T10:03:07.456 INFO:teuthology.provision.fog.smithi041:Waiting for deploy to finish 2021-05-15T10:03:07.526 INFO:teuthology.provision.fog.smithi033:Waiting for deploy to finish 2021-05-15T10:07:26.611 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2021-05-15T10:07:42.889 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-15T10:07:53.212 INFO:teuthology.provision.fog.smithi041:Node is ready 2021-05-15T10:07:53.231 INFO:teuthology.orchestra.run.smithi041.stdout:smithi041.front.sepia.ceph.com 2021-05-15T10:07:53.288 INFO:teuthology.orchestra.run.smithi041.stdout:172.21.15.41 smithi041.front.sepia.ceph.com smithi041 2021-05-15T10:07:53.628 INFO:teuthology.provision.fog.smithi041:Deploy complete! 2021-05-15T10:08:58.385 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-15T10:09:14.164 INFO:teuthology.provision.fog.smithi033:Node is ready 2021-05-15T10:09:14.184 INFO:teuthology.orchestra.run.smithi033.stdout:smithi033.front.sepia.ceph.com 2021-05-15T10:09:14.242 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33 smithi033.front.sepia.ceph.com smithi033 2021-05-15T10:09:14.595 INFO:teuthology.provision.fog.smithi033:Deploy complete! 2021-05-15T10:09:14.738 INFO:teuthology.lock.ops:Checking smithi033.front.sepia.ceph.com 2021-05-15T10:09:14.739 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-15T10:09:14.755 INFO:teuthology.lock.ops:Checking smithi041.front.sepia.ceph.com 2021-05-15T10:09:14.756 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-15T10:09:14.770 INFO:teuthology.lock.ops:Updating [smithi033.front.sepia.ceph.com]: set os type and version on server 2021-05-15T10:09:14.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-15T10:09:15.043 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2021-05-15T10:09:15.097 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="CentOS Linux" 2021-05-15T10:09:15.097 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="8 (Core)" 2021-05-15T10:09:15.097 INFO:teuthology.orchestra.run.smithi033.stdout:ID="centos" 2021-05-15T10:09:15.098 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE="rhel fedora" 2021-05-15T10:09:15.098 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="8" 2021-05-15T10:09:15.098 INFO:teuthology.orchestra.run.smithi033.stdout:PLATFORM_ID="platform:el8" 2021-05-15T10:09:15.099 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-05-15T10:09:15.099 INFO:teuthology.orchestra.run.smithi033.stdout:ANSI_COLOR="0;31" 2021-05-15T10:09:15.099 INFO:teuthology.orchestra.run.smithi033.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-15T10:09:15.099 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://www.centos.org/" 2021-05-15T10:09:15.100 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-15T10:09:15.100 INFO:teuthology.orchestra.run.smithi033.stdout: 2021-05-15T10:09:15.100 INFO:teuthology.orchestra.run.smithi033.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-15T10:09:15.100 INFO:teuthology.orchestra.run.smithi033.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-15T10:09:15.100 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-05-15T10:09:15.101 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-05-15T10:09:15.101 INFO:teuthology.orchestra.run.smithi033.stdout: 2021-05-15T10:09:15.102 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2021-05-15T10:09:15.117 INFO:teuthology.lock.ops:Updating [smithi041.front.sepia.ceph.com]: set os type and version on server 2021-05-15T10:09:15.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-15T10:09:15.336 INFO:teuthology.orchestra.run.smithi041.stdout:x86_64 2021-05-15T10:09:15.392 INFO:teuthology.orchestra.run.smithi041.stdout:NAME="CentOS Linux" 2021-05-15T10:09:15.392 INFO:teuthology.orchestra.run.smithi041.stdout:VERSION="8 (Core)" 2021-05-15T10:09:15.392 INFO:teuthology.orchestra.run.smithi041.stdout:ID="centos" 2021-05-15T10:09:15.393 INFO:teuthology.orchestra.run.smithi041.stdout:ID_LIKE="rhel fedora" 2021-05-15T10:09:15.393 INFO:teuthology.orchestra.run.smithi041.stdout:VERSION_ID="8" 2021-05-15T10:09:15.393 INFO:teuthology.orchestra.run.smithi041.stdout:PLATFORM_ID="platform:el8" 2021-05-15T10:09:15.393 INFO:teuthology.orchestra.run.smithi041.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-05-15T10:09:15.394 INFO:teuthology.orchestra.run.smithi041.stdout:ANSI_COLOR="0;31" 2021-05-15T10:09:15.394 INFO:teuthology.orchestra.run.smithi041.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-15T10:09:15.394 INFO:teuthology.orchestra.run.smithi041.stdout:HOME_URL="https://www.centos.org/" 2021-05-15T10:09:15.394 INFO:teuthology.orchestra.run.smithi041.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-15T10:09:15.395 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-05-15T10:09:15.395 INFO:teuthology.orchestra.run.smithi041.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-15T10:09:15.395 INFO:teuthology.orchestra.run.smithi041.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-15T10:09:15.395 INFO:teuthology.orchestra.run.smithi041.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-05-15T10:09:15.395 INFO:teuthology.orchestra.run.smithi041.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-05-15T10:09:15.396 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-05-15T10:09:15.396 INFO:teuthology.lock.ops:Updating smithi041.front.sepia.ceph.com on lock server 2021-05-15T10:09:15.499 INFO:teuthology.dispatcher.supervisor:Running job 6115955 2021-05-15T10:09:15.634 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/virtualenv/bin/teuthology -v --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2021-05-15_10:00:36-rados-wip-kefu-testing-2021-05-15-1545-distro-basic-smithi/6115955 --name kchai-2021-05-15_10:00:36-rados-wip-kefu-testing-2021-05-15-1545-distro-basic-smithi --description rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} -- /home/teuthworker/archive/kchai-2021-05-15_10:00:36-rados-wip-kefu-testing-2021-05-15-1545-distro-basic-smithi/6115955/orig.config.yaml 2021-05-15T10:09:15.637 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/kchai-2021-05-15_10:00:36-rados-wip-kefu-testing-2021-05-15-1545-distro-basic-smithi/6115955 2021-05-15T10:09:15.637 INFO:teuthology.dispatcher.supervisor:Job PID: 12096 2021-05-15T10:09:15.638 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-15T22:09:33.379 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-15T22:09:33.559 INFO:teuthology.kill:Killing Pids: {12096} 2021-05-15T22:09:34.583 INFO:teuthology.kill:Nuking machines: ['smithi033', 'smithi041'] 2021-05-15T22:09:35.466 INFO:teuthology.kill:2021-05-15 22:09:35,465.465 INFO:teuthology.nuke:targets: 2021-05-15T22:09:35.467 INFO:teuthology.kill: smithi033.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgJH2R4ue7PF1lgH4/3mdx7xHDLYB3gKew5e/jw64J0wD45yOKycsaRsn5lMKiZJ1ciqlBoOo2E2+E0uV5+D9CDuVyLyy0Br3anjzehKXzbEleWz3mLKIEiY/2GMjHr1Iy4CQbh81f7VHJs7RjDHRO4ERvroB8W6sp91eKv9OLcNMITyLq3eCvJXfDAI+T+qMJgSA0ZRSBDzrPZoHIHVvbUnBpsVfvJUIP8j71QM2YqseXAxqShq4PtCgB/JtiWgzFkB0W3hrUcF8GxR+TJxNzfZQtuZdk9dfXCe3SkFa56/S9Vm73oe/jxfOMrzL2DFthN7A5i3XM8MILL9SezQYh7FJD5npXfI1gk7eHVho9tyd5qKbm1qfkQbIHYvy3U9kx1/PWVmX/Q6MIuJcuMHdgQUduJNBtZfO7zDbgQFvS2VLEYlgTmOwwtM9Y/RityWyT7uTgEKrDuiaVguiNxWCv+HITIhywliybZ4j9sAMoGYJ7kTSTpNI0a8qSTCdIkm8= 2021-05-15T22:09:35.467 INFO:teuthology.kill: smithi041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxyoS+jN+jGAJAnluAd8meQXVFP2Fu+xZEEXhN78xrhTIb0YDrPQIefH/C6UE+VdRrWJP0bLuLiDzS1+hIeAQj9uV0bL2SvjN1PnYr6iGWVtXKlSVz3Hp85WB704nM6zXmTxunQWRgCFxvNIHTIZHbM1J2F81Sf1a67gjwZjcoC1LwXweFURg+U7E0IoTG57EGw09l4+UEOLhgZb0GDgd9vYqzj91D5EL7gLvwZXCMy2oQkMKkBL5nI73ui6Zz96RTfjsWlsi1ncnG4m7K5E3Tdu35nfc3Jp1zwkoQLXwAXJzwa2ZOe/sbGkAdKfDgCTKegfE13sGBp4VquN0Z79QcTw+HL8vkYmQWb1z32df0aRRz7HlIT3PAZkVkIggQ3nECiIfwX3jwYnm0GXJ5i+xvEKJnNMC3UBnoEuG87V9YjQ7LZaYVE45isCwRbzEGwUOpOddw5F4qMBEAjtM6hezJVOfcXx32MPBLMs6VQpnRxwa4ZE+0OoMsH6gl/XRrn50= 2021-05-15T22:09:35.467 INFO:teuthology.kill:2021-05-15 22:09:35,466.466 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-15T22:09:35.468 INFO:teuthology.kill:2021-05-15 22:09:35,468.468 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-15T22:09:35.523 INFO:teuthology.kill:2021-05-15 22:09:35,523.523 INFO:teuthology.orchestra.console:Power off smithi033 2021-05-15T22:09:35.535 INFO:teuthology.kill:2021-05-15 22:09:35,534.534 INFO:teuthology.orchestra.console:Power off smithi041 2021-05-15T22:09:43.896 INFO:teuthology.kill:2021-05-15 22:09:43,896.896 INFO:teuthology.orchestra.console:Power off for smithi041 completed 2021-05-15T22:09:48.003 INFO:teuthology.kill:2021-05-15 22:09:48,003.003 INFO:teuthology.orchestra.console:Power off for smithi033 completed 2021-05-15T22:09:48.197 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2021-05-15T22:09:48.198 INFO:teuthology.task.internal:roles: ubuntu@smithi041.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2021-05-15T22:09:48.198 INFO:teuthology.misc:Compressing logs... 2021-05-15T22:09:48.199 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-15T22:10:06.667 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-15T22:10:06.713 INFO:teuthology.kill:No teuthology processes running 2021-05-15T22:10:07.731 INFO:teuthology.kill:Nuking machines: ['smithi033', 'smithi041'] 2021-05-15T22:10:08.639 INFO:teuthology.kill:2021-05-15 22:10:08,639.639 INFO:teuthology.nuke:targets: 2021-05-15T22:10:08.640 INFO:teuthology.kill: smithi033.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgJH2R4ue7PF1lgH4/3mdx7xHDLYB3gKew5e/jw64J0wD45yOKycsaRsn5lMKiZJ1ciqlBoOo2E2+E0uV5+D9CDuVyLyy0Br3anjzehKXzbEleWz3mLKIEiY/2GMjHr1Iy4CQbh81f7VHJs7RjDHRO4ERvroB8W6sp91eKv9OLcNMITyLq3eCvJXfDAI+T+qMJgSA0ZRSBDzrPZoHIHVvbUnBpsVfvJUIP8j71QM2YqseXAxqShq4PtCgB/JtiWgzFkB0W3hrUcF8GxR+TJxNzfZQtuZdk9dfXCe3SkFa56/S9Vm73oe/jxfOMrzL2DFthN7A5i3XM8MILL9SezQYh7FJD5npXfI1gk7eHVho9tyd5qKbm1qfkQbIHYvy3U9kx1/PWVmX/Q6MIuJcuMHdgQUduJNBtZfO7zDbgQFvS2VLEYlgTmOwwtM9Y/RityWyT7uTgEKrDuiaVguiNxWCv+HITIhywliybZ4j9sAMoGYJ7kTSTpNI0a8qSTCdIkm8= 2021-05-15T22:10:08.640 INFO:teuthology.kill: smithi041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxyoS+jN+jGAJAnluAd8meQXVFP2Fu+xZEEXhN78xrhTIb0YDrPQIefH/C6UE+VdRrWJP0bLuLiDzS1+hIeAQj9uV0bL2SvjN1PnYr6iGWVtXKlSVz3Hp85WB704nM6zXmTxunQWRgCFxvNIHTIZHbM1J2F81Sf1a67gjwZjcoC1LwXweFURg+U7E0IoTG57EGw09l4+UEOLhgZb0GDgd9vYqzj91D5EL7gLvwZXCMy2oQkMKkBL5nI73ui6Zz96RTfjsWlsi1ncnG4m7K5E3Tdu35nfc3Jp1zwkoQLXwAXJzwa2ZOe/sbGkAdKfDgCTKegfE13sGBp4VquN0Z79QcTw+HL8vkYmQWb1z32df0aRRz7HlIT3PAZkVkIggQ3nECiIfwX3jwYnm0GXJ5i+xvEKJnNMC3UBnoEuG87V9YjQ7LZaYVE45isCwRbzEGwUOpOddw5F4qMBEAjtM6hezJVOfcXx32MPBLMs6VQpnRxwa4ZE+0OoMsH6gl/XRrn50= 2021-05-15T22:10:08.660 INFO:teuthology.kill:2021-05-15 22:10:08,660.660 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-15T22:10:08.663 INFO:teuthology.kill:2021-05-15 22:10:08,663.663 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-15T22:10:08.717 INFO:teuthology.kill:2021-05-15 22:10:08,717.717 INFO:teuthology.orchestra.console:Power off smithi033 2021-05-15T22:10:08.730 INFO:teuthology.kill:2021-05-15 22:10:08,730.730 INFO:teuthology.orchestra.console:Power off smithi041 2021-05-15T22:10:08.859 INFO:teuthology.kill:2021-05-15 22:10:08,858.858 INFO:teuthology.orchestra.console:Power off for smithi033 completed 2021-05-15T22:10:08.871 INFO:teuthology.kill:2021-05-15 22:10:08,871.871 INFO:teuthology.orchestra.console:Power off for smithi041 completed 2021-05-15T22:10:09.001 INFO:teuthology.kill:2021-05-15 22:10:09,001.001 INFO:teuthology.lock.ops:unlocked smithi033.front.sepia.ceph.com 2021-05-15T22:10:09.007 INFO:teuthology.kill:2021-05-15 22:10:09,007.007 INFO:teuthology.lock.ops:unlocked smithi041.front.sepia.ceph.com 2021-05-15T22:12:09.165 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15