2021-05-01T05:30:48.570 INFO:root:teuthology version: 1.1.0-bea5b739 2021-05-01T05:30:48.678 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi019.front.sepia.ceph.com' reimaging 2021-05-01T05:30:48.678 INFO:teuthology.lock.ops:Updating [ubuntu@smithi019.front.sepia.ceph.com]: reset os type and version on server 2021-05-01T05:30:48.678 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2021-05-01T05:30:48.695 INFO:teuthology.lock.ops:Node 'ubuntu@smithi019.front.sepia.ceph.com' reimaging is complete 2021-05-01T05:30:48.695 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi102.front.sepia.ceph.com' reimaging 2021-05-01T05:30:48.695 INFO:teuthology.lock.ops:Updating [ubuntu@smithi102.front.sepia.ceph.com]: reset os type and version on server 2021-05-01T05:30:48.696 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2021-05-01T05:30:48.752 INFO:teuthology.lock.ops:Node 'ubuntu@smithi102.front.sepia.ceph.com' reimaging is complete 2021-05-01T05:30:48.897 INFO:teuthology.provision.fog.smithi019:Scheduling deploy of centos 8.3 2021-05-01T05:30:48.958 INFO:teuthology.provision.fog.smithi102:Scheduling deploy of centos 8.3 2021-05-01T05:30:49.289 INFO:teuthology.orchestra.console:Power off smithi019 2021-05-01T05:30:49.307 INFO:teuthology.orchestra.console:Power off smithi102 2021-05-01T05:30:57.690 INFO:teuthology.orchestra.console:Power off for smithi102 completed 2021-05-01T05:30:57.695 INFO:teuthology.orchestra.console:Power off for smithi019 completed 2021-05-01T05:30:57.791 INFO:teuthology.orchestra.console:Power on smithi102 2021-05-01T05:30:57.799 INFO:teuthology.orchestra.console:Power on smithi019 2021-05-01T05:31:02.060 INFO:teuthology.orchestra.console:Power on for smithi019 completed 2021-05-01T05:31:02.161 INFO:teuthology.provision.fog.smithi019:Waiting for deploy to finish 2021-05-01T05:31:06.167 INFO:teuthology.orchestra.console:Power on for smithi102 completed 2021-05-01T05:31:06.267 INFO:teuthology.provision.fog.smithi102:Waiting for deploy to finish 2021-05-01T05:35:18.909 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-01T05:35:19.251 INFO:teuthology.provision.fog.smithi019:Node is ready 2021-05-01T05:35:19.266 INFO:teuthology.orchestra.run.smithi019.stdout:smithi019.front.sepia.ceph.com 2021-05-01T05:35:19.322 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19 smithi019.front.sepia.ceph.com smithi019 2021-05-01T05:35:19.657 INFO:teuthology.provision.fog.smithi019:Deploy complete! 2021-05-01T05:35:21.436 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-01T05:35:41.844 INFO:teuthology.provision.fog.smithi102:Node is ready 2021-05-01T05:35:41.863 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102.front.sepia.ceph.com 2021-05-01T05:35:41.918 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102 smithi102.front.sepia.ceph.com smithi102 2021-05-01T05:35:42.251 INFO:teuthology.provision.fog.smithi102:Deploy complete! 2021-05-01T05:35:42.382 INFO:teuthology.lock.ops:Checking smithi019.front.sepia.ceph.com 2021-05-01T05:35:42.382 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-01T05:35:42.399 INFO:teuthology.lock.ops:Checking smithi102.front.sepia.ceph.com 2021-05-01T05:35:42.399 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-01T05:35:42.413 INFO:teuthology.lock.ops:Updating [smithi019.front.sepia.ceph.com]: set os type and version on server 2021-05-01T05:35:42.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-01T05:35:42.709 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2021-05-01T05:35:42.764 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="CentOS Linux" 2021-05-01T05:35:42.764 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="8" 2021-05-01T05:35:42.764 INFO:teuthology.orchestra.run.smithi019.stdout:ID="centos" 2021-05-01T05:35:42.764 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="rhel fedora" 2021-05-01T05:35:42.764 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="8" 2021-05-01T05:35:42.764 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el8" 2021-05-01T05:35:42.764 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="CentOS Linux 8" 2021-05-01T05:35:42.764 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31" 2021-05-01T05:35:42.765 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-01T05:35:42.765 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://centos.org/" 2021-05-01T05:35:42.765 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-01T05:35:42.765 INFO:teuthology.orchestra.run.smithi019.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-01T05:35:42.765 INFO:teuthology.orchestra.run.smithi019.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-01T05:35:42.766 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2021-05-01T05:35:42.782 INFO:teuthology.lock.ops:Updating [smithi102.front.sepia.ceph.com]: set os type and version on server 2021-05-01T05:35:42.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-01T05:35:43.071 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2021-05-01T05:35:43.129 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="CentOS Linux" 2021-05-01T05:35:43.129 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="8" 2021-05-01T05:35:43.129 INFO:teuthology.orchestra.run.smithi102.stdout:ID="centos" 2021-05-01T05:35:43.130 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="rhel fedora" 2021-05-01T05:35:43.130 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="8" 2021-05-01T05:35:43.130 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el8" 2021-05-01T05:35:43.130 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="CentOS Linux 8" 2021-05-01T05:35:43.130 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2021-05-01T05:35:43.130 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-01T05:35:43.130 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://centos.org/" 2021-05-01T05:35:43.130 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-01T05:35:43.130 INFO:teuthology.orchestra.run.smithi102.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-01T05:35:43.130 INFO:teuthology.orchestra.run.smithi102.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-01T05:35:43.131 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2021-05-01T05:35:43.207 INFO:teuthology.dispatcher.supervisor:Running job 6082172 2021-05-01T05:35:43.207 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_bea5b7399b9dcb9d756a32fdfc32f498319d425f/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2021-04-29_13:53:41-rados-benhanokh-NCB-distro-basic-smithi/6082172 --name benhanokh-2021-04-29_13:53:41-rados-benhanokh-NCB-distro-basic-smithi --description rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} -- /home/teuthworker/archive/benhanokh-2021-04-29_13:53:41-rados-benhanokh-NCB-distro-basic-smithi/6082172/orig.config.yaml 2021-05-01T05:35:43.213 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2021-04-29_13:53:41-rados-benhanokh-NCB-distro-basic-smithi/6082172 2021-05-01T05:35:43.214 INFO:teuthology.dispatcher.supervisor:Job PID: 3161 2021-05-01T05:35:43.214 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-01T17:36:34.091 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-01T17:36:34.290 INFO:teuthology.kill:Killing Pids: {3161} 2021-05-01T17:36:35.362 INFO:teuthology.kill:Nuking machines: ['smithi019', 'smithi102'] 2021-05-01T17:36:36.297 INFO:teuthology.kill:2021-05-01 17:36:36,297.297 INFO:teuthology.nuke:targets: 2021-05-01T17:36:36.298 INFO:teuthology.kill: smithi019.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtTo9Tl7oKf/xoqhqr6NMCfJ+x9cLhr1ySQv/R9IJEaWlE5NwX4lKtOoC7p19KTBzUe214+y66xk839Zht11eX+vDi0UqcOhKA0nGBOiDHZ9gnT2hDVFgC8rWyRp6BZT0VVwv9JtwKg9Qh9jG72onS1+KEE5DLs6/D0bYy/DqGJq/42eXo5u/oYqykz0ONf7ukiHkrCSUnpg0P1Oo/+lg8xOKPhBb8OHmkARyJwe+ILs/UlAX3OF/04APuLGbulT9hFyeMDExJn6xpTAZVSR4elrQ+QLD+hgRXIrjDyYJxfCNvRYJ1cJSUsMkDrRJrrafNPW67LeHozYWTxSur897bEpaCIQGVnQOnrAEPeRSrW3Dm97oWYVJo1JEtvDOqptNibbUcpeoounltJJaYy2FJ6qCiCILqrAOI0Lbc8TrQrg8O80bzO9V7xOsOoWvmoITxnN/l4Z6tXBEhP4L9zMjfAHkwK9pAkFprHXgSI9O8NpDB2Qsy5Gml+e8I4rjg9NE= 2021-05-01T17:36:36.298 INFO:teuthology.kill: smithi102.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdG1blSQVcLuikRsfSMAIcwggpTskI6KlSPYWGY0ZgOjcwBjtuVsVZOfLiCD5aJBdDNlCrLtL/pK71hS1UPi3+cJXzKMdpWuDLQ2eu8w/oiHgh9x7AhO1JdiKTTh11TxLanGeOwqb+Vms31UFwKY+BXcJv3SG9LrYcpETJw0kptlvUs04bgnE4R4PkZiQLedyjwlb0xyPitcVvonFXI8QOHIgCu9F5N0hM7ElC1ov6yXFcMChApvLQTeLGMvfbnjUkoHqOVH1pw+xXWvD8ev4gQWDh2+DFAFA0P2+/BVjdKw6NFiz+7NsWRvhO1lGaEhe3KwBP4xwtmG2Tb4qoHf5ZRKbwtKhv/GLh0q18cpSliCaGmHXTMwqvsJfEexPBRJrnxUG1JSRJpC2ItkxIuqHAa5Kp8RlK1AStBiFt+ngJDzWZI0aDmQcHep4cc3Bm5G2rHJY2U1Ur1FvkCyVs4pWM6naJjyh7SzwFCfG3ghnkOTKlIgzMjZgBhuNbIoyzT78= 2021-05-01T17:36:36.298 INFO:teuthology.kill:2021-05-01 17:36:36,297.297 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-01T17:36:36.300 INFO:teuthology.kill:2021-05-01 17:36:36,300.300 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-01T17:36:36.360 INFO:teuthology.kill:2021-05-01 17:36:36,360.360 INFO:teuthology.orchestra.console:Power off smithi019 2021-05-01T17:36:36.373 INFO:teuthology.kill:2021-05-01 17:36:36,373.373 INFO:teuthology.orchestra.console:Power off smithi102 2021-05-01T17:36:48.846 INFO:teuthology.kill:2021-05-01 17:36:48,845.845 INFO:teuthology.orchestra.console:Power off for smithi019 completed 2021-05-01T17:36:48.850 INFO:teuthology.kill:2021-05-01 17:36:48,850.850 INFO:teuthology.orchestra.console:Power off for smithi102 completed 2021-05-01T17:36:49.075 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-05-01T17:36:49.075 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-05-01T17:36:49.075 INFO:teuthology.misc:Compressing logs... 2021-05-01T17:36:49.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-01T17:37:49.135 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_bea5b7399b9dcb9d756a32fdfc32f498319d425f/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_bea5b7399b9dcb9d756a32fdfc32f498319d425f/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_bea5b7399b9dcb9d756a32fdfc32f498319d425f/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_bea5b7399b9dcb9d756a32fdfc32f498319d425f/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_bea5b7399b9dcb9d756a32fdfc32f498319d425f/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_bea5b7399b9dcb9d756a32fdfc32f498319d425f/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_bea5b7399b9dcb9d756a32fdfc32f498319d425f/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-01T17:37:49.199 INFO:teuthology.kill:No teuthology processes running 2021-05-01T17:37:50.257 INFO:teuthology.kill:Nuking machines: ['smithi019', 'smithi102'] 2021-05-01T17:37:51.177 INFO:teuthology.kill:2021-05-01 17:37:51,177.177 INFO:teuthology.nuke:targets: 2021-05-01T17:37:51.178 INFO:teuthology.kill: smithi019.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtTo9Tl7oKf/xoqhqr6NMCfJ+x9cLhr1ySQv/R9IJEaWlE5NwX4lKtOoC7p19KTBzUe214+y66xk839Zht11eX+vDi0UqcOhKA0nGBOiDHZ9gnT2hDVFgC8rWyRp6BZT0VVwv9JtwKg9Qh9jG72onS1+KEE5DLs6/D0bYy/DqGJq/42eXo5u/oYqykz0ONf7ukiHkrCSUnpg0P1Oo/+lg8xOKPhBb8OHmkARyJwe+ILs/UlAX3OF/04APuLGbulT9hFyeMDExJn6xpTAZVSR4elrQ+QLD+hgRXIrjDyYJxfCNvRYJ1cJSUsMkDrRJrrafNPW67LeHozYWTxSur897bEpaCIQGVnQOnrAEPeRSrW3Dm97oWYVJo1JEtvDOqptNibbUcpeoounltJJaYy2FJ6qCiCILqrAOI0Lbc8TrQrg8O80bzO9V7xOsOoWvmoITxnN/l4Z6tXBEhP4L9zMjfAHkwK9pAkFprHXgSI9O8NpDB2Qsy5Gml+e8I4rjg9NE= 2021-05-01T17:37:51.178 INFO:teuthology.kill: smithi102.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdG1blSQVcLuikRsfSMAIcwggpTskI6KlSPYWGY0ZgOjcwBjtuVsVZOfLiCD5aJBdDNlCrLtL/pK71hS1UPi3+cJXzKMdpWuDLQ2eu8w/oiHgh9x7AhO1JdiKTTh11TxLanGeOwqb+Vms31UFwKY+BXcJv3SG9LrYcpETJw0kptlvUs04bgnE4R4PkZiQLedyjwlb0xyPitcVvonFXI8QOHIgCu9F5N0hM7ElC1ov6yXFcMChApvLQTeLGMvfbnjUkoHqOVH1pw+xXWvD8ev4gQWDh2+DFAFA0P2+/BVjdKw6NFiz+7NsWRvhO1lGaEhe3KwBP4xwtmG2Tb4qoHf5ZRKbwtKhv/GLh0q18cpSliCaGmHXTMwqvsJfEexPBRJrnxUG1JSRJpC2ItkxIuqHAa5Kp8RlK1AStBiFt+ngJDzWZI0aDmQcHep4cc3Bm5G2rHJY2U1Ur1FvkCyVs4pWM6naJjyh7SzwFCfG3ghnkOTKlIgzMjZgBhuNbIoyzT78= 2021-05-01T17:37:51.199 INFO:teuthology.kill:2021-05-01 17:37:51,199.199 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-01T17:37:51.202 INFO:teuthology.kill:2021-05-01 17:37:51,202.202 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-01T17:37:51.256 INFO:teuthology.kill:2021-05-01 17:37:51,256.256 INFO:teuthology.orchestra.console:Power off smithi102 2021-05-01T17:37:51.275 INFO:teuthology.kill:2021-05-01 17:37:51,275.275 INFO:teuthology.orchestra.console:Power off smithi019 2021-05-01T17:37:51.403 INFO:teuthology.kill:2021-05-01 17:37:51,403.403 INFO:teuthology.orchestra.console:Power off for smithi102 completed 2021-05-01T17:37:51.414 INFO:teuthology.kill:2021-05-01 17:37:51,414.414 INFO:teuthology.orchestra.console:Power off for smithi019 completed 2021-05-01T17:37:51.554 INFO:teuthology.kill:2021-05-01 17:37:51,553.553 INFO:teuthology.lock.ops:unlocked smithi102.front.sepia.ceph.com 2021-05-01T17:37:51.556 INFO:teuthology.kill:2021-05-01 17:37:51,556.556 INFO:teuthology.lock.ops:unlocked smithi019.front.sepia.ceph.com 2021-05-01T17:39:51.825 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-05-01T17:39:51.919 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-05-01T17:39:51.919 INFO:teuthology.nuke:Checking targets against current locks 2021-05-01T17:39:51.999 INFO:teuthology.nuke:Not nuking smithi019.front.sepia.ceph.com because description doesn't match 2021-05-01T17:39:51.999 INFO:teuthology.nuke:Not nuking smithi102.front.sepia.ceph.com because description doesn't match