2021-02-12T22:52:22.392 INFO:root:teuthology version: 1.1.0-9ec73f3a 2021-02-12T22:52:22.501 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi001.front.sepia.ceph.com' reimaging 2021-02-12T22:52:22.501 INFO:teuthology.lock.ops:Updating [ubuntu@smithi001.front.sepia.ceph.com]: reset os type and version on server 2021-02-12T22:52:22.501 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2021-02-12T22:52:22.517 INFO:teuthology.lock.ops:Node 'ubuntu@smithi001.front.sepia.ceph.com' reimaging is complete 2021-02-12T22:52:22.517 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi007.front.sepia.ceph.com' reimaging 2021-02-12T22:52:22.517 INFO:teuthology.lock.ops:Updating [ubuntu@smithi007.front.sepia.ceph.com]: reset os type and version on server 2021-02-12T22:52:22.517 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2021-02-12T22:52:22.534 INFO:teuthology.lock.ops:Node 'ubuntu@smithi007.front.sepia.ceph.com' reimaging is complete 2021-02-12T22:52:22.736 INFO:teuthology.provision.fog.smithi001:Scheduling deploy of centos 8.2 2021-02-12T22:52:22.747 INFO:teuthology.provision.fog.smithi007:Scheduling deploy of centos 8.2 2021-02-12T22:52:23.190 INFO:teuthology.orchestra.console:Power off smithi007 2021-02-12T22:52:23.225 INFO:teuthology.orchestra.console:Power off smithi001 2021-02-12T22:52:31.558 INFO:teuthology.orchestra.console:Power off for smithi007 completed 2021-02-12T22:52:31.591 INFO:teuthology.orchestra.console:Power off for smithi001 completed 2021-02-12T22:52:31.660 INFO:teuthology.orchestra.console:Power on smithi007 2021-02-12T22:52:31.696 INFO:teuthology.orchestra.console:Power on smithi001 2021-02-12T22:52:40.029 INFO:teuthology.orchestra.console:Power on for smithi007 completed 2021-02-12T22:52:40.060 INFO:teuthology.orchestra.console:Power on for smithi001 completed 2021-02-12T22:52:40.131 INFO:teuthology.provision.fog.smithi007:Waiting for deploy to finish 2021-02-12T22:52:40.161 INFO:teuthology.provision.fog.smithi001:Waiting for deploy to finish 2021-02-12T22:56:29.893 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-12T22:56:41.036 INFO:teuthology.provision.fog.smithi007:Node is ready 2021-02-12T22:56:41.055 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007.front.sepia.ceph.com 2021-02-12T22:56:41.114 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7 smithi007.front.sepia.ceph.com smithi007 2021-02-12T22:56:41.472 INFO:teuthology.provision.fog.smithi007:Deploy complete! 2021-02-12T22:56:44.482 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-02-12T22:56:47.984 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-12T22:56:48.325 INFO:teuthology.provision.fog.smithi001:Node is ready 2021-02-12T22:56:48.340 INFO:teuthology.orchestra.run.smithi001.stdout:smithi001.front.sepia.ceph.com 2021-02-12T22:56:48.395 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1 smithi001.front.sepia.ceph.com smithi001 2021-02-12T22:56:48.722 INFO:teuthology.provision.fog.smithi001:Deploy complete! 2021-02-12T22:56:48.847 INFO:teuthology.lock.ops:Checking smithi001.front.sepia.ceph.com 2021-02-12T22:56:48.847 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-12T22:56:48.862 INFO:teuthology.lock.ops:Checking smithi007.front.sepia.ceph.com 2021-02-12T22:56:48.862 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-12T22:56:48.876 INFO:teuthology.lock.ops:Updating [smithi001.front.sepia.ceph.com]: set os type and version on server 2021-02-12T22:56:48.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-12T22:56:49.150 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2021-02-12T22:56:49.205 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="CentOS Linux" 2021-02-12T22:56:49.205 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="8 (Core)" 2021-02-12T22:56:49.205 INFO:teuthology.orchestra.run.smithi001.stdout:ID="centos" 2021-02-12T22:56:49.205 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE="rhel fedora" 2021-02-12T22:56:49.205 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="8" 2021-02-12T22:56:49.205 INFO:teuthology.orchestra.run.smithi001.stdout:PLATFORM_ID="platform:el8" 2021-02-12T22:56:49.205 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-02-12T22:56:49.205 INFO:teuthology.orchestra.run.smithi001.stdout:ANSI_COLOR="0;31" 2021-02-12T22:56:49.205 INFO:teuthology.orchestra.run.smithi001.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-02-12T22:56:49.205 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://www.centos.org/" 2021-02-12T22:56:49.206 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-02-12T22:56:49.206 INFO:teuthology.orchestra.run.smithi001.stdout: 2021-02-12T22:56:49.206 INFO:teuthology.orchestra.run.smithi001.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-02-12T22:56:49.206 INFO:teuthology.orchestra.run.smithi001.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-02-12T22:56:49.206 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-02-12T22:56:49.206 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-02-12T22:56:49.206 INFO:teuthology.orchestra.run.smithi001.stdout: 2021-02-12T22:56:49.207 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2021-02-12T22:56:49.222 INFO:teuthology.lock.ops:Updating [smithi007.front.sepia.ceph.com]: set os type and version on server 2021-02-12T22:56:49.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-12T22:56:49.501 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2021-02-12T22:56:49.556 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="CentOS Linux" 2021-02-12T22:56:49.556 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="8 (Core)" 2021-02-12T22:56:49.556 INFO:teuthology.orchestra.run.smithi007.stdout:ID="centos" 2021-02-12T22:56:49.556 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="rhel fedora" 2021-02-12T22:56:49.556 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="8" 2021-02-12T22:56:49.556 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el8" 2021-02-12T22:56:49.556 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-02-12T22:56:49.556 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2021-02-12T22:56:49.556 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-02-12T22:56:49.556 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://www.centos.org/" 2021-02-12T22:56:49.557 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-02-12T22:56:49.557 INFO:teuthology.orchestra.run.smithi007.stdout: 2021-02-12T22:56:49.557 INFO:teuthology.orchestra.run.smithi007.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-02-12T22:56:49.557 INFO:teuthology.orchestra.run.smithi007.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-02-12T22:56:49.557 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-02-12T22:56:49.557 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-02-12T22:56:49.557 INFO:teuthology.orchestra.run.smithi007.stdout: 2021-02-12T22:56:49.557 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2021-02-12T22:56:49.637 INFO:teuthology.dispatcher.supervisor:Running job 5874219 2021-02-12T22:56:49.637 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_9ec73f3aa0a18603afa55289d00653272f224a69/virtualenv/bin/teuthology -v --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2021-02-10_23:48:41-rados-wip-sage2-testing-2021-02-10-1604-distro-basic-smithi/5874219 --name sage-2021-02-10_23:48:41-rados-wip-sage2-testing-2021-02-10-1604-distro-basic-smithi --description rados/cephadm/thrash/{0-distro/centos_latest 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} -- /home/teuthworker/archive/sage-2021-02-10_23:48:41-rados-wip-sage2-testing-2021-02-10-1604-distro-basic-smithi/5874219/orig.config.yaml 2021-02-12T22:56:49.645 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/sage-2021-02-10_23:48:41-rados-wip-sage2-testing-2021-02-10-1604-distro-basic-smithi/5874219 2021-02-12T22:56:49.646 INFO:teuthology.dispatcher.supervisor:Job PID: 28536 2021-02-12T22:56:49.646 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-02-13T10:57:06.138 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-02-13T10:57:06.551 INFO:teuthology.kill:Killing Pids: {28536} 2021-02-13T10:57:07.429 INFO:teuthology.kill:Nuking machines: ['smithi001', 'smithi007'] 2021-02-13T10:57:08.171 INFO:teuthology.kill:2021-02-13 10:57:08,170.170 INFO:teuthology.nuke:targets: 2021-02-13T10:57:08.171 INFO:teuthology.kill: smithi001.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGS0u8AOnNSJTdO/QdE944/OmYe+zOM9VXNisjbFzxmFnRzWFRuBWcVRG68RBbqB+U3aHxUV1t7/qNutWAqwcQcasP+ODcqWVzWObgWs3BK7P8/wz2o4utcbb+YRO+PsmyMoXG1cXW5YJfZr9G6y+t0mA3tr+M1jW5UfiycwU3Z7w/35KdfkcCI6NZhehH6EFoxXBuEAfuQmkD4l8LWB7/Pcr6h2ETnHW3ajVKAglnaQyoqaJYH67OsHV6aq0/NHPCKKijPhp9DMsDV61lywis1Rh95+2JXqvJ9v146Tt87YvYiHkG2mMJjJzlbdluIGQlBEv1BgRtUP/YF57ck0PqwN7aHTj3NLfbYh7waIKcv1vmf6XcVMkUYTmkvlV5TmcAmBqnIY0UJ4fn+TvvlHuYhyFXRNpMLKoAxmDs78/qKoVz56gKlSusSBlP3lN1ipNs5UyjOxyvCrXhabZIBIg3oam26DgpBMkHyLeHHukMm7Tph9Z9M88tsl0akbMaY3c= 2021-02-13T10:57:08.171 INFO:teuthology.kill: smithi007.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCesZCAcalA1tfQjGc610af9IgfQbz/QXvCKrkqXsG2QyD9IxBpdRe2J2PVpjLjEMhzKCPF52YUDw5j2Sv64L70RuLTMpbkyOQT8npkB92UPG7Ro58lYoLMSMSINrX85retggMkbM81M+VOfohOw8dpGxfmwOSaqZo+vTltK/I5HqGoDVMDQBwvtCH6nClzObEshsVc2DWTtFFzk/YK9BLHDTWAZqmYwvqPF5q8y3bGracJ+tg6M4uHvOE31FFKuEnz3dwUSVCZbghNMlaN3PYSnFMG7LGr1hGuQmHyVl/TQztWoXm3+LVZDhfH8mLduI+avd9XUm7tb0oumLRdNaUXLxVcDdXfF1Rk9PT4DS36kzdQif23PJvoI66sWIxik4vc/2X3EnaRzygsONMV+zFm98IeLRAeGurA2S2WlOsOw58kv2hoho61H0RKpvGOkc2+KL+AM19+I4IlMWF5GwDbtfsazDzOpHJI5gnh4DVXzl4tbPMZdeJhmQwzP16QNpM= 2021-02-13T10:57:08.171 INFO:teuthology.kill:2021-02-13 10:57:08,171.171 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-13T10:57:08.177 INFO:teuthology.kill:2021-02-13 10:57:08,176.176 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-13T10:57:08.235 INFO:teuthology.kill:2021-02-13 10:57:08,234.234 INFO:teuthology.orchestra.console:Power off smithi007 2021-02-13T10:57:08.247 INFO:teuthology.kill:2021-02-13 10:57:08,247.247 INFO:teuthology.orchestra.console:Power off smithi001 2021-02-13T10:57:16.602 INFO:teuthology.kill:2021-02-13 10:57:16,602.602 INFO:teuthology.orchestra.console:Power off for smithi007 completed 2021-02-13T10:57:20.719 INFO:teuthology.kill:2021-02-13 10:57:20,719.719 INFO:teuthology.orchestra.console:Power off for smithi001 completed 2021-02-13T10:57:20.950 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.realm.zone.a', 'node-exporter.a', 'alertmanager.a'] 2021-02-13T10:57:20.950 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2021-02-13T10:57:20.951 INFO:teuthology.misc:Compressing logs... 2021-02-13T10:57:20.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-13T10:58:20.955 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_9ec73f3aa0a18603afa55289d00653272f224a69/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9ec73f3aa0a18603afa55289d00653272f224a69/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9ec73f3aa0a18603afa55289d00653272f224a69/teuthology/misc.py", line 1372, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_9ec73f3aa0a18603afa55289d00653272f224a69/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_9ec73f3aa0a18603afa55289d00653272f224a69/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_9ec73f3aa0a18603afa55289d00653272f224a69/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9ec73f3aa0a18603afa55289d00653272f224a69/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-02-13T10:58:21.004 INFO:teuthology.kill:No teuthology processes running 2021-02-13T10:58:21.864 INFO:teuthology.kill:Nuking machines: ['smithi001', 'smithi007'] 2021-02-13T10:58:22.599 INFO:teuthology.kill:2021-02-13 10:58:22,599.599 INFO:teuthology.nuke:targets: 2021-02-13T10:58:22.600 INFO:teuthology.kill: smithi001.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGS0u8AOnNSJTdO/QdE944/OmYe+zOM9VXNisjbFzxmFnRzWFRuBWcVRG68RBbqB+U3aHxUV1t7/qNutWAqwcQcasP+ODcqWVzWObgWs3BK7P8/wz2o4utcbb+YRO+PsmyMoXG1cXW5YJfZr9G6y+t0mA3tr+M1jW5UfiycwU3Z7w/35KdfkcCI6NZhehH6EFoxXBuEAfuQmkD4l8LWB7/Pcr6h2ETnHW3ajVKAglnaQyoqaJYH67OsHV6aq0/NHPCKKijPhp9DMsDV61lywis1Rh95+2JXqvJ9v146Tt87YvYiHkG2mMJjJzlbdluIGQlBEv1BgRtUP/YF57ck0PqwN7aHTj3NLfbYh7waIKcv1vmf6XcVMkUYTmkvlV5TmcAmBqnIY0UJ4fn+TvvlHuYhyFXRNpMLKoAxmDs78/qKoVz56gKlSusSBlP3lN1ipNs5UyjOxyvCrXhabZIBIg3oam26DgpBMkHyLeHHukMm7Tph9Z9M88tsl0akbMaY3c= 2021-02-13T10:58:22.600 INFO:teuthology.kill: smithi007.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCesZCAcalA1tfQjGc610af9IgfQbz/QXvCKrkqXsG2QyD9IxBpdRe2J2PVpjLjEMhzKCPF52YUDw5j2Sv64L70RuLTMpbkyOQT8npkB92UPG7Ro58lYoLMSMSINrX85retggMkbM81M+VOfohOw8dpGxfmwOSaqZo+vTltK/I5HqGoDVMDQBwvtCH6nClzObEshsVc2DWTtFFzk/YK9BLHDTWAZqmYwvqPF5q8y3bGracJ+tg6M4uHvOE31FFKuEnz3dwUSVCZbghNMlaN3PYSnFMG7LGr1hGuQmHyVl/TQztWoXm3+LVZDhfH8mLduI+avd9XUm7tb0oumLRdNaUXLxVcDdXfF1Rk9PT4DS36kzdQif23PJvoI66sWIxik4vc/2X3EnaRzygsONMV+zFm98IeLRAeGurA2S2WlOsOw58kv2hoho61H0RKpvGOkc2+KL+AM19+I4IlMWF5GwDbtfsazDzOpHJI5gnh4DVXzl4tbPMZdeJhmQwzP16QNpM= 2021-02-13T10:58:22.621 INFO:teuthology.kill:2021-02-13 10:58:22,621.621 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-13T10:58:22.624 INFO:teuthology.kill:2021-02-13 10:58:22,624.624 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-13T10:58:22.678 INFO:teuthology.kill:2021-02-13 10:58:22,678.678 INFO:teuthology.orchestra.console:Power off smithi001 2021-02-13T10:58:22.690 INFO:teuthology.kill:2021-02-13 10:58:22,690.690 INFO:teuthology.orchestra.console:Power off smithi007 2021-02-13T10:58:22.818 INFO:teuthology.kill:2021-02-13 10:58:22,818.818 INFO:teuthology.orchestra.console:Power off for smithi001 completed 2021-02-13T10:58:22.831 INFO:teuthology.kill:2021-02-13 10:58:22,831.831 INFO:teuthology.orchestra.console:Power off for smithi007 completed 2021-02-13T10:58:22.962 INFO:teuthology.kill:2021-02-13 10:58:22,961.961 INFO:teuthology.lock.ops:unlocked smithi001.front.sepia.ceph.com 2021-02-13T10:58:22.968 INFO:teuthology.kill:2021-02-13 10:58:22,968.968 INFO:teuthology.lock.ops:unlocked smithi007.front.sepia.ceph.com 2021-02-13T11:00:23.145 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-02-13T11:00:23.235 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-02-13T11:00:23.235 INFO:teuthology.nuke:Checking targets against current locks 2021-02-13T11:00:23.325 INFO:teuthology.nuke:Not nuking smithi007.front.sepia.ceph.com because description doesn't match 2021-02-13T11:00:23.325 INFO:teuthology.nuke:Not nuking smithi001.front.sepia.ceph.com because description doesn't match