2021-08-27T08:46:15.897 INFO:root:teuthology version: 1.1.0-d0ccb5e7 2021-08-27T08:46:16.012 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi116.front.sepia.ceph.com' reimaging 2021-08-27T08:46:16.013 INFO:teuthology.lock.ops:Updating [ubuntu@smithi116.front.sepia.ceph.com]: reset os type and version on server 2021-08-27T08:46:16.014 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2021-08-27T08:46:16.033 INFO:teuthology.lock.ops:Node 'ubuntu@smithi116.front.sepia.ceph.com' reimaging is complete 2021-08-27T08:46:16.034 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi163.front.sepia.ceph.com' reimaging 2021-08-27T08:46:16.034 INFO:teuthology.lock.ops:Updating [ubuntu@smithi163.front.sepia.ceph.com]: reset os type and version on server 2021-08-27T08:46:16.034 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2021-08-27T08:46:16.054 INFO:teuthology.lock.ops:Node 'ubuntu@smithi163.front.sepia.ceph.com' reimaging is complete 2021-08-27T08:46:16.251 INFO:teuthology.provision.fog.smithi116:Scheduling deploy of centos 8.3 2021-08-27T08:46:16.254 INFO:teuthology.provision.fog.smithi163:Scheduling deploy of centos 8.3 2021-08-27T08:46:16.819 INFO:teuthology.orchestra.console:Power off smithi163 2021-08-27T08:46:16.842 INFO:teuthology.orchestra.console:Power off smithi116 2021-08-27T08:46:25.223 INFO:teuthology.orchestra.console:Power off for smithi163 completed 2021-08-27T08:46:25.233 INFO:teuthology.orchestra.console:Power off for smithi116 completed 2021-08-27T08:46:25.325 INFO:teuthology.orchestra.console:Power on smithi163 2021-08-27T08:46:25.346 INFO:teuthology.orchestra.console:Power on smithi116 2021-08-27T08:46:33.713 INFO:teuthology.orchestra.console:Power on for smithi163 completed 2021-08-27T08:46:33.801 INFO:teuthology.orchestra.console:Power on for smithi116 completed 2021-08-27T08:46:33.824 INFO:teuthology.provision.fog.smithi163:Waiting for deploy to finish 2021-08-27T08:46:33.902 INFO:teuthology.provision.fog.smithi116:Waiting for deploy to finish 2021-08-27T08:51:14.885 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-27T08:51:15.185 INFO:teuthology.provision.fog.smithi116:Node is ready 2021-08-27T08:51:15.201 INFO:teuthology.orchestra.run.smithi116.stdout:smithi116.front.sepia.ceph.com 2021-08-27T08:51:15.256 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116 smithi116.front.sepia.ceph.com smithi116 2021-08-27T08:51:15.574 INFO:teuthology.provision.fog.smithi116:Deploy complete! 2021-08-27T08:51:48.064 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-27T08:51:48.362 INFO:teuthology.provision.fog.smithi163:Node is ready 2021-08-27T08:51:48.377 INFO:teuthology.orchestra.run.smithi163.stdout:smithi163.front.sepia.ceph.com 2021-08-27T08:51:48.433 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.15.163 smithi163.front.sepia.ceph.com smithi163 2021-08-27T08:51:48.766 INFO:teuthology.provision.fog.smithi163:Deploy complete! 2021-08-27T08:51:48.959 INFO:teuthology.lock.ops:Checking smithi116.front.sepia.ceph.com 2021-08-27T08:51:48.960 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-27T08:51:48.982 INFO:teuthology.lock.ops:Checking smithi163.front.sepia.ceph.com 2021-08-27T08:51:48.982 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-27T08:51:49.003 INFO:teuthology.lock.ops:Updating [smithi116.front.sepia.ceph.com]: set os type and version on server 2021-08-27T08:51:49.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-27T08:51:49.320 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2021-08-27T08:51:49.376 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Linux" 2021-08-27T08:51:49.376 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="8" 2021-08-27T08:51:49.376 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2021-08-27T08:51:49.377 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2021-08-27T08:51:49.377 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="8" 2021-08-27T08:51:49.378 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el8" 2021-08-27T08:51:49.378 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Linux 8" 2021-08-27T08:51:49.379 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2021-08-27T08:51:49.379 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-27T08:51:49.379 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2021-08-27T08:51:49.381 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-08-27T08:51:49.381 INFO:teuthology.orchestra.run.smithi116.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-08-27T08:51:49.382 INFO:teuthology.orchestra.run.smithi116.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-08-27T08:51:49.383 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2021-08-27T08:51:49.415 INFO:teuthology.lock.ops:Updating [smithi163.front.sepia.ceph.com]: set os type and version on server 2021-08-27T08:51:49.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-27T08:51:49.624 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2021-08-27T08:51:49.679 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Linux" 2021-08-27T08:51:49.680 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="8" 2021-08-27T08:51:49.680 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos" 2021-08-27T08:51:49.680 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora" 2021-08-27T08:51:49.681 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="8" 2021-08-27T08:51:49.681 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el8" 2021-08-27T08:51:49.681 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Linux 8" 2021-08-27T08:51:49.681 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2021-08-27T08:51:49.682 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-27T08:51:49.682 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://centos.org/" 2021-08-27T08:51:49.682 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-08-27T08:51:49.682 INFO:teuthology.orchestra.run.smithi163.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-08-27T08:51:49.683 INFO:teuthology.orchestra.run.smithi163.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-08-27T08:51:49.683 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2021-08-27T08:51:49.771 INFO:teuthology.dispatcher.supervisor:Running job 6362012 2021-08-27T08:51:49.772 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/bin/teuthology -v --owner scheduled_ideepika@teuthology --archive /home/teuthworker/archive/ideepika-2021-08-27_07:09:34-rados-wip-deepika2-testing-2021-08-27-0331-distro-basic-smithi/6362012 --name ideepika-2021-08-27_07:09:34-rados-wip-deepika2-testing-2021-08-27-0331-distro-basic-smithi --description rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} -- /home/teuthworker/archive/ideepika-2021-08-27_07:09:34-rados-wip-deepika2-testing-2021-08-27-0331-distro-basic-smithi/6362012/orig.config.yaml 2021-08-27T08:51:49.776 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/ideepika-2021-08-27_07:09:34-rados-wip-deepika2-testing-2021-08-27-0331-distro-basic-smithi/6362012 2021-08-27T08:51:49.776 INFO:teuthology.dispatcher.supervisor:Job PID: 8368 2021-08-27T08:51:49.777 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-08-27T20:52:45.691 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-08-27T20:52:45.908 INFO:teuthology.kill:Killing Pids: {8368} 2021-08-27T20:52:47.171 INFO:teuthology.kill:Nuking machines: ['smithi116', 'smithi163'] 2021-08-27T20:52:48.193 INFO:teuthology.kill:2021-08-27 20:52:48,192.192 INFO:teuthology.nuke:targets: 2021-08-27T20:52:48.194 INFO:teuthology.kill: smithi116.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZyUAdNZ5OMwnWdclNzEaT1zPU6wz0Rzo6PcqhCOvqJQ8UfX+8bxn98okpki+R6ns1skNCSjaalZ4eFrsQS9xEdXX3HsjdUpXuCsyfqFBfr6c1EyOTftk54mD2qGRzTF91EqzizcrEUIvu4qi02J1wnd/cugK5YxKuBItQs5M6jv4Yfczuo4SnpwzH3fiHf+B7zpVrLdVWKSB+nrEBIJ4JaJgK/XO/X4QqNO0CqdxVyiOTSfQJqhuGcHXtPOgNxk5Llf4nVQ5JZoPjKuZvvXbtygRBfAZhVjgvcEKFUXRByFV9cv5aGyWsSU80PIFrROwhw+TnwYeemn/8THn+rFnb2Du8BVqGOwjKHWWCQ/Fm0b76olFdPHJlXPXsqbPFuMRZwi49PZkzoyzWj7lPVNGFJDsaF3K3JiC9WJcj1DYqNH22JrlzYC0huShO9u2FQTDA5/et78CzGAKlvoBEBpWqj74Pq2RCkhCaKrLmviJ8dUXAqHq7tsiO2FFHdGO1t/E= 2021-08-27T20:52:48.197 INFO:teuthology.kill: smithi163.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoRUswoZtdbuXU3C/zT6vyfjzGrTkcBJrACRDsSU7CDJYMd1sWfLL1/RVPXMopeGNcCCv+tXFVQmolQrVsI1N+IBTbueXCFdHILf9s+T39Pan9DY/3sa2jjlnd72W2YXun13mO+f1d0c26puDNRryo5iVJgQaEU9PqVxUL9Tux67lpJypyGvE4PkSd3f4iX1ex1ozeu0vXrEHS3D2rtGydRph/YmJagqrGb+R0nyW31fjhMhgP21vNL0zRgpkDPoeN/sGmuyoRSN0zT001RpMowxLFCwwwUZJC8W4HMra++2TB/VzN2YWnOD0WvBAHmsWKSjwfZW9A4KhawboG9aeJZNrFf9mO6vmx1zzvR+JVdJptqdpCx27l7Odynk41apXTHiSPTp2pSx2UKYm5KpMFvh416fwphz0PMRYNUJ1w+jbysx/b2uquuBLK50AEGGfAN3YlTQifuoJhKruiTmpJ3UFYrvgukbHhF//B4gHtdNciFU1K7JTOgbVruCWRrXE= 2021-08-27T20:52:48.197 INFO:teuthology.kill:2021-08-27 20:52:48,193.193 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-27T20:52:48.197 INFO:teuthology.kill:2021-08-27 20:52:48,196.196 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-27T20:52:48.263 INFO:teuthology.kill:2021-08-27 20:52:48,263.263 INFO:teuthology.orchestra.console:Power off smithi116 2021-08-27T20:52:48.279 INFO:teuthology.kill:2021-08-27 20:52:48,278.278 INFO:teuthology.orchestra.console:Power off smithi163 2021-08-27T20:52:56.636 INFO:teuthology.kill:2021-08-27 20:52:56,635.635 INFO:teuthology.orchestra.console:Power off for smithi116 completed 2021-08-27T20:53:00.755 INFO:teuthology.kill:2021-08-27 20:53:00,755.755 INFO:teuthology.orchestra.console:Power off for smithi163 completed 2021-08-27T20:53:00.990 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-08-27T20:53:00.991 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-08-27T20:53:00.991 INFO:teuthology.misc:Compressing logs... 2021-08-27T20:53:00.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-27T20:54:01.046 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-08-27T20:54:01.108 INFO:teuthology.kill:No teuthology processes running 2021-08-27T20:54:02.259 INFO:teuthology.kill:Nuking machines: ['smithi116', 'smithi163'] 2021-08-27T20:54:03.294 INFO:teuthology.kill:2021-08-27 20:54:03,294.294 INFO:teuthology.nuke:targets: 2021-08-27T20:54:03.295 INFO:teuthology.kill: smithi116.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZyUAdNZ5OMwnWdclNzEaT1zPU6wz0Rzo6PcqhCOvqJQ8UfX+8bxn98okpki+R6ns1skNCSjaalZ4eFrsQS9xEdXX3HsjdUpXuCsyfqFBfr6c1EyOTftk54mD2qGRzTF91EqzizcrEUIvu4qi02J1wnd/cugK5YxKuBItQs5M6jv4Yfczuo4SnpwzH3fiHf+B7zpVrLdVWKSB+nrEBIJ4JaJgK/XO/X4QqNO0CqdxVyiOTSfQJqhuGcHXtPOgNxk5Llf4nVQ5JZoPjKuZvvXbtygRBfAZhVjgvcEKFUXRByFV9cv5aGyWsSU80PIFrROwhw+TnwYeemn/8THn+rFnb2Du8BVqGOwjKHWWCQ/Fm0b76olFdPHJlXPXsqbPFuMRZwi49PZkzoyzWj7lPVNGFJDsaF3K3JiC9WJcj1DYqNH22JrlzYC0huShO9u2FQTDA5/et78CzGAKlvoBEBpWqj74Pq2RCkhCaKrLmviJ8dUXAqHq7tsiO2FFHdGO1t/E= 2021-08-27T20:54:03.295 INFO:teuthology.kill: smithi163.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoRUswoZtdbuXU3C/zT6vyfjzGrTkcBJrACRDsSU7CDJYMd1sWfLL1/RVPXMopeGNcCCv+tXFVQmolQrVsI1N+IBTbueXCFdHILf9s+T39Pan9DY/3sa2jjlnd72W2YXun13mO+f1d0c26puDNRryo5iVJgQaEU9PqVxUL9Tux67lpJypyGvE4PkSd3f4iX1ex1ozeu0vXrEHS3D2rtGydRph/YmJagqrGb+R0nyW31fjhMhgP21vNL0zRgpkDPoeN/sGmuyoRSN0zT001RpMowxLFCwwwUZJC8W4HMra++2TB/VzN2YWnOD0WvBAHmsWKSjwfZW9A4KhawboG9aeJZNrFf9mO6vmx1zzvR+JVdJptqdpCx27l7Odynk41apXTHiSPTp2pSx2UKYm5KpMFvh416fwphz0PMRYNUJ1w+jbysx/b2uquuBLK50AEGGfAN3YlTQifuoJhKruiTmpJ3UFYrvgukbHhF//B4gHtdNciFU1K7JTOgbVruCWRrXE= 2021-08-27T20:54:03.317 INFO:teuthology.kill:2021-08-27 20:54:03,317.317 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-27T20:54:03.321 INFO:teuthology.kill:2021-08-27 20:54:03,321.321 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-27T20:54:03.375 INFO:teuthology.kill:2021-08-27 20:54:03,375.375 INFO:teuthology.orchestra.console:Power off smithi163 2021-08-27T20:54:03.388 INFO:teuthology.kill:2021-08-27 20:54:03,388.388 INFO:teuthology.orchestra.console:Power off smithi116 2021-08-27T20:54:03.518 INFO:teuthology.kill:2021-08-27 20:54:03,518.518 INFO:teuthology.orchestra.console:Power off for smithi163 completed 2021-08-27T20:54:03.531 INFO:teuthology.kill:2021-08-27 20:54:03,531.531 INFO:teuthology.orchestra.console:Power off for smithi116 completed 2021-08-27T20:54:03.662 INFO:teuthology.kill:2021-08-27 20:54:03,661.661 INFO:teuthology.lock.ops:unlocked smithi163.front.sepia.ceph.com 2021-08-27T20:54:03.673 INFO:teuthology.kill:2021-08-27 20:54:03,672.672 INFO:teuthology.lock.ops:unlocked smithi116.front.sepia.ceph.com 2021-08-27T20:56:03.970 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-08-27T20:56:04.061 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-08-27T20:56:04.061 INFO:teuthology.nuke:Checking targets against current locks 2021-08-27T20:56:04.149 INFO:teuthology.nuke:Not nuking smithi163.front.sepia.ceph.com because description doesn't match 2021-08-27T20:56:04.149 INFO:teuthology.nuke:Not nuking smithi116.front.sepia.ceph.com because description doesn't match