2021-05-13T19:27:54.732 INFO:root:teuthology version: 1.1.0-19220a3b 2021-05-13T19:27:54.906 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi032.front.sepia.ceph.com' reimaging 2021-05-13T19:27:54.907 INFO:teuthology.lock.ops:Updating [ubuntu@smithi032.front.sepia.ceph.com]: reset os type and version on server 2021-05-13T19:27:54.907 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2021-05-13T19:27:54.940 INFO:teuthology.lock.ops:Node 'ubuntu@smithi032.front.sepia.ceph.com' reimaging is complete 2021-05-13T19:27:54.940 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi106.front.sepia.ceph.com' reimaging 2021-05-13T19:27:54.941 INFO:teuthology.lock.ops:Updating [ubuntu@smithi106.front.sepia.ceph.com]: reset os type and version on server 2021-05-13T19:27:54.941 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server 2021-05-13T19:27:54.960 INFO:teuthology.lock.ops:Node 'ubuntu@smithi106.front.sepia.ceph.com' reimaging is complete 2021-05-13T19:27:54.961 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi133.front.sepia.ceph.com' reimaging 2021-05-13T19:27:54.961 INFO:teuthology.lock.ops:Updating [ubuntu@smithi133.front.sepia.ceph.com]: reset os type and version on server 2021-05-13T19:27:54.961 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server 2021-05-13T19:27:54.979 INFO:teuthology.lock.ops:Node 'ubuntu@smithi133.front.sepia.ceph.com' reimaging is complete 2021-05-13T19:27:54.979 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi190.front.sepia.ceph.com' reimaging 2021-05-13T19:27:54.980 INFO:teuthology.lock.ops:Updating [ubuntu@smithi190.front.sepia.ceph.com]: reset os type and version on server 2021-05-13T19:27:54.980 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2021-05-13T19:27:54.996 INFO:teuthology.lock.ops:Node 'ubuntu@smithi190.front.sepia.ceph.com' reimaging is complete 2021-05-13T19:27:55.145 INFO:teuthology.provision.fog.smithi032:Scheduling deploy of rhel 8.3 2021-05-13T19:27:55.167 INFO:teuthology.provision.fog.smithi106:Scheduling deploy of rhel 8.3 2021-05-13T19:27:55.191 INFO:teuthology.provision.fog.smithi133:Scheduling deploy of rhel 8.3 2021-05-13T19:27:55.220 INFO:teuthology.provision.fog.smithi190:Scheduling deploy of rhel 8.3 2021-05-13T19:27:55.459 INFO:teuthology.orchestra.console:Power off smithi032 2021-05-13T19:27:55.496 INFO:teuthology.orchestra.console:Power off smithi106 2021-05-13T19:27:55.532 INFO:teuthology.orchestra.console:Power off smithi133 2021-05-13T19:27:55.588 INFO:teuthology.orchestra.console:Power off smithi190 2021-05-13T19:28:03.896 INFO:teuthology.orchestra.console:Power off for smithi133 completed 2021-05-13T19:28:03.954 INFO:teuthology.orchestra.console:Power off for smithi190 completed 2021-05-13T19:28:03.998 INFO:teuthology.orchestra.console:Power on smithi133 2021-05-13T19:28:04.057 INFO:teuthology.orchestra.console:Power on smithi190 2021-05-13T19:28:07.949 INFO:teuthology.orchestra.console:Power off for smithi032 completed 2021-05-13T19:28:07.976 INFO:teuthology.orchestra.console:Power off for smithi106 completed 2021-05-13T19:28:08.056 INFO:teuthology.orchestra.console:Power on smithi032 2021-05-13T19:28:08.081 INFO:teuthology.orchestra.console:Power on smithi106 2021-05-13T19:28:12.312 INFO:teuthology.orchestra.console:Power on for smithi032 completed 2021-05-13T19:28:12.361 INFO:teuthology.orchestra.console:Power on for smithi106 completed 2021-05-13T19:28:12.414 INFO:teuthology.provision.fog.smithi032:Waiting for deploy to finish 2021-05-13T19:28:12.427 INFO:teuthology.orchestra.console:Power on for smithi190 completed 2021-05-13T19:28:12.447 INFO:teuthology.orchestra.console:Power on for smithi133 completed 2021-05-13T19:28:12.462 INFO:teuthology.provision.fog.smithi106:Waiting for deploy to finish 2021-05-13T19:28:12.531 INFO:teuthology.provision.fog.smithi190:Waiting for deploy to finish 2021-05-13T19:28:12.549 INFO:teuthology.provision.fog.smithi133:Waiting for deploy to finish 2021-05-13T19:30:43.918 ERROR:teuthology.orchestra.connection:Error authenticating with smithi106.front.sepia.ceph.com: Authentication failed. 2021-05-13T19:32:12.110 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-13T19:32:19.041 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-13T19:32:32.539 INFO:teuthology.provision.fog.smithi133:Node is ready 2021-05-13T19:32:32.560 INFO:teuthology.orchestra.run.smithi133.stdout:smithi133.front.sepia.ceph.com 2021-05-13T19:32:32.617 INFO:teuthology.orchestra.run.smithi133.stdout:172.21.15.133 smithi133.front.sepia.ceph.com smithi133 2021-05-13T19:32:32.976 INFO:teuthology.provision.fog.smithi133:Deploy complete! 2021-05-13T19:32:35.408 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-13T19:32:35.704 INFO:teuthology.provision.fog.smithi032:Node is ready 2021-05-13T19:32:35.719 INFO:teuthology.orchestra.run.smithi032.stdout:smithi032.front.sepia.ceph.com 2021-05-13T19:32:35.774 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32 smithi032.front.sepia.ceph.com smithi032 2021-05-13T19:32:36.099 INFO:teuthology.provision.fog.smithi032:Deploy complete! 2021-05-13T19:32:39.101 INFO:teuthology.provision.fog.smithi190:Node is ready 2021-05-13T19:32:39.120 INFO:teuthology.orchestra.run.smithi190.stdout:smithi190.front.sepia.ceph.com 2021-05-13T19:32:39.178 INFO:teuthology.orchestra.run.smithi190.stdout:172.21.15.190 smithi190.front.sepia.ceph.com smithi190 2021-05-13T19:32:39.542 INFO:teuthology.provision.fog.smithi190:Deploy complete! 2021-05-13T19:32:57.178 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-13T19:32:57.503 INFO:teuthology.provision.fog.smithi106:Node is ready 2021-05-13T19:32:57.518 INFO:teuthology.orchestra.run.smithi106.stdout:smithi106.front.sepia.ceph.com 2021-05-13T19:32:57.573 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106 smithi106.front.sepia.ceph.com smithi106 2021-05-13T19:32:57.915 INFO:teuthology.provision.fog.smithi106:Deploy complete! 2021-05-13T19:32:58.079 INFO:teuthology.lock.ops:Checking smithi032.front.sepia.ceph.com 2021-05-13T19:32:58.080 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-13T19:32:58.096 INFO:teuthology.lock.ops:Checking smithi106.front.sepia.ceph.com 2021-05-13T19:32:58.096 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-13T19:32:58.109 INFO:teuthology.lock.ops:Checking smithi133.front.sepia.ceph.com 2021-05-13T19:32:58.109 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-13T19:32:58.123 INFO:teuthology.lock.ops:Checking smithi190.front.sepia.ceph.com 2021-05-13T19:32:58.123 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-13T19:32:58.137 INFO:teuthology.lock.ops:Updating [smithi032.front.sepia.ceph.com]: set os type and version on server 2021-05-13T19:32:58.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-13T19:32:58.368 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2021-05-13T19:32:58.424 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="Red Hat Enterprise Linux" 2021-05-13T19:32:58.424 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="8.3 (Ootpa)" 2021-05-13T19:32:58.424 INFO:teuthology.orchestra.run.smithi032.stdout:ID="rhel" 2021-05-13T19:32:58.425 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="fedora" 2021-05-13T19:32:58.425 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="8.3" 2021-05-13T19:32:58.425 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el8" 2021-05-13T19:32:58.425 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-05-13T19:32:58.425 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2021-05-13T19:32:58.426 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-05-13T19:32:58.426 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://www.redhat.com/" 2021-05-13T19:32:58.426 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-05-13T19:32:58.426 INFO:teuthology.orchestra.run.smithi032.stdout: 2021-05-13T19:32:58.427 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-05-13T19:32:58.427 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-05-13T19:32:58.427 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-05-13T19:32:58.427 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-05-13T19:32:58.428 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2021-05-13T19:32:58.443 INFO:teuthology.lock.ops:Updating [smithi106.front.sepia.ceph.com]: set os type and version on server 2021-05-13T19:32:58.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-13T19:32:58.656 INFO:teuthology.orchestra.run.smithi106.stdout:x86_64 2021-05-13T19:32:58.710 INFO:teuthology.orchestra.run.smithi106.stdout:NAME="Red Hat Enterprise Linux" 2021-05-13T19:32:58.710 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION="8.3 (Ootpa)" 2021-05-13T19:32:58.710 INFO:teuthology.orchestra.run.smithi106.stdout:ID="rhel" 2021-05-13T19:32:58.711 INFO:teuthology.orchestra.run.smithi106.stdout:ID_LIKE="fedora" 2021-05-13T19:32:58.711 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_ID="8.3" 2021-05-13T19:32:58.711 INFO:teuthology.orchestra.run.smithi106.stdout:PLATFORM_ID="platform:el8" 2021-05-13T19:32:58.711 INFO:teuthology.orchestra.run.smithi106.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-05-13T19:32:58.712 INFO:teuthology.orchestra.run.smithi106.stdout:ANSI_COLOR="0;31" 2021-05-13T19:32:58.712 INFO:teuthology.orchestra.run.smithi106.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-05-13T19:32:58.712 INFO:teuthology.orchestra.run.smithi106.stdout:HOME_URL="https://www.redhat.com/" 2021-05-13T19:32:58.713 INFO:teuthology.orchestra.run.smithi106.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-05-13T19:32:58.713 INFO:teuthology.orchestra.run.smithi106.stdout: 2021-05-13T19:32:58.713 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-05-13T19:32:58.713 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-05-13T19:32:58.714 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-05-13T19:32:58.714 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-05-13T19:32:58.714 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server 2021-05-13T19:32:58.729 INFO:teuthology.lock.ops:Updating [smithi133.front.sepia.ceph.com]: set os type and version on server 2021-05-13T19:32:58.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-13T19:32:59.016 INFO:teuthology.orchestra.run.smithi133.stdout:x86_64 2021-05-13T19:32:59.071 INFO:teuthology.orchestra.run.smithi133.stdout:NAME="Red Hat Enterprise Linux" 2021-05-13T19:32:59.071 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION="8.3 (Ootpa)" 2021-05-13T19:32:59.072 INFO:teuthology.orchestra.run.smithi133.stdout:ID="rhel" 2021-05-13T19:32:59.072 INFO:teuthology.orchestra.run.smithi133.stdout:ID_LIKE="fedora" 2021-05-13T19:32:59.072 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_ID="8.3" 2021-05-13T19:32:59.073 INFO:teuthology.orchestra.run.smithi133.stdout:PLATFORM_ID="platform:el8" 2021-05-13T19:32:59.073 INFO:teuthology.orchestra.run.smithi133.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-05-13T19:32:59.073 INFO:teuthology.orchestra.run.smithi133.stdout:ANSI_COLOR="0;31" 2021-05-13T19:32:59.073 INFO:teuthology.orchestra.run.smithi133.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-05-13T19:32:59.074 INFO:teuthology.orchestra.run.smithi133.stdout:HOME_URL="https://www.redhat.com/" 2021-05-13T19:32:59.074 INFO:teuthology.orchestra.run.smithi133.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-05-13T19:32:59.074 INFO:teuthology.orchestra.run.smithi133.stdout: 2021-05-13T19:32:59.074 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-05-13T19:32:59.075 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-05-13T19:32:59.075 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-05-13T19:32:59.076 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-05-13T19:32:59.077 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server 2021-05-13T19:32:59.092 INFO:teuthology.lock.ops:Updating [smithi190.front.sepia.ceph.com]: set os type and version on server 2021-05-13T19:32:59.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-13T19:32:59.380 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2021-05-13T19:32:59.433 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="Red Hat Enterprise Linux" 2021-05-13T19:32:59.433 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="8.3 (Ootpa)" 2021-05-13T19:32:59.433 INFO:teuthology.orchestra.run.smithi190.stdout:ID="rhel" 2021-05-13T19:32:59.434 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="fedora" 2021-05-13T19:32:59.434 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="8.3" 2021-05-13T19:32:59.434 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el8" 2021-05-13T19:32:59.434 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-05-13T19:32:59.435 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2021-05-13T19:32:59.435 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-05-13T19:32:59.435 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://www.redhat.com/" 2021-05-13T19:32:59.435 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-05-13T19:32:59.436 INFO:teuthology.orchestra.run.smithi190.stdout: 2021-05-13T19:32:59.436 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-05-13T19:32:59.436 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-05-13T19:32:59.436 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-05-13T19:32:59.436 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-05-13T19:32:59.437 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2021-05-13T19:32:59.518 INFO:teuthology.dispatcher.supervisor:Running job 6113676 2021-05-13T19:32:59.518 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2021-05-13_15:35:24-rados-no_column_b-distro-basic-smithi/6113676 --name benhanokh-2021-05-13_15:35:24-rados-no_column_b-distro-basic-smithi --description rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} -- /home/teuthworker/archive/benhanokh-2021-05-13_15:35:24-rados-no_column_b-distro-basic-smithi/6113676/orig.config.yaml 2021-05-13T19:32:59.522 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2021-05-13_15:35:24-rados-no_column_b-distro-basic-smithi/6113676 2021-05-13T19:32:59.522 INFO:teuthology.dispatcher.supervisor:Job PID: 8243 2021-05-13T19:32:59.523 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-14T07:33:16.443 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-14T07:33:16.647 INFO:teuthology.kill:Killing Pids: {8243} 2021-05-14T07:33:17.805 INFO:teuthology.kill:Nuking machines: ['smithi032', 'smithi106', 'smithi133', 'smithi190'] 2021-05-14T07:33:18.784 INFO:teuthology.kill:2021-05-14 07:33:18,784.784 INFO:teuthology.nuke:targets: 2021-05-14T07:33:18.785 INFO:teuthology.kill: smithi032.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQsfW6AfwbWfXYpF6MuEvXVKMRmINVjwvldGS4gM8It9AuB+MeWHbNHtGAwPIDAL6nCPfimH3h/MhQCBgPvNIV6jr8qwHGLBQ8QUgYF5Xx+0A4zIqYd0UW3wngHdf8AogiPelUS8J3cdFxBW1QVVxZMzCDkpLa5mlg3FYA3EvQNqzpv1s7tSgXfQ998pFmfddEKVT/mbipz2HVCXXz5j5ljiyoRzewbpPbkaz3MD3/Olb4M5bKqo6OUFSvA3L6/HK1lgd8TasqOITH1mNxt5uUTswmrrG4Eq7+Go4txXtshC0uGimoPUZA4vT9McnvVucx8oBNSEuGp1jMOn/Vd9LY0e5Ach5AmO+tOeDYzWogzN9pkl/YahV152LGYkosa1rAhbaaVLKOjQaaC3VxY9dnNr+PlPHIS879g9QTvsg+s8BNzNHzpXhIZlDJtpGPjvaoirfVqtQ7owCu7+01pWjdKRmGGXkwPj3jsTJlj9iYgx1kuXp/An0dpvgvQXAgqu0= 2021-05-14T07:33:18.785 INFO:teuthology.kill: smithi106.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8f8K7QRcYFmnSWc8p8HXZ9FdlUnv/Rm35q7YJS1YsQ/UKM6veoVPf3TUlEdHYnulVoodG33zv1tYrQzbj7vcV2FZ4K3JBU0gVEnD1JBokvZcYGeW1Xj3Qx2xoNE0oJ0MyUTQ+aY96EEYS9szVx2cTch/BeE4ZWQXGKMLWIjtMM5r8G8x889pCWc1NGCjpGvCIlfTW3tsmEOAL5zOoa1lumKVY7FYbJJQAgZPr+vim+M3GDRTkyt5d2Qj6rfEx2bv+VGZX+65LRj8VYOgpOCQkoCOySRyirAFaJzrooCILvTBSrB3pcVPsiSdBCIRw+0vLS6gtqxNbF5dUAwgdQyjW3DPiHisjcWz2lvirWXnNPhpYHnzMB/zUbDs+aMvpD+c82S7mtc2vSd5fvWIXktjfCBUeaB4XRQevnFRD8alHqRPAK3mM2oVeZblnszKGq23gFNRyaPxwjq6EuGQPnaueN2kmf/jWEn/nyIZ3NJH5EGB/QaS8oaP2qlimFqb4Wnc= 2021-05-14T07:33:18.785 INFO:teuthology.kill: smithi133.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvr1uYsE05Mo0/5zMMQJ9Lp+fDpWU1HyBV6x4HisXPRETIgH6/Tp7z7bd/8RaIMEVAY+NRXrBn7tj1u0TD36949S0bXA/3mRR4epT0nWOiksfYYTzmF66g9Qscz/+O5wv8z2G8kAn015F0FKGHnX2UzBYcwDdFrI7cqgrhTQg1TOGmrbwFjarY0H4S6zXVLQ/kkYymJ2ncf/Rh6Pfzm1ZROWbA+kyhBY07O+hABakPMsvVLhfE2usFMVpfYdwUiPeoDc/nX7AmcwBsEQ+TvkwDCi+8WCqzEAn5dGhNmt4R+DN4ujHvqQi3ulQJojGo2wXE9R+keqo/PL1wVb8nwACouyyT/ts45YEJR86PZLkNc3ojzjDuTFCV6VngE5b2ApsmTmJONBQh46Pn3HMr6kb0dr8X1OWWFN1vRlGn0K7JqU/MvCzeljk2rjUVTLhI2LVMYFohNMANdOWrNI7I/ZAnwfMVMIDOoNNB8o36+WZyHQmdcPg4MU7rviBzfnEBZSU= 2021-05-14T07:33:18.786 INFO:teuthology.kill: smithi190.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCHJKMm12aMLp+Gjd3zicYfa6EDvcY8Y0OJbw0pqKP0JRsiWNHiGCCnEVR5QgRQsEJ7NStLEFSjtQVzknoxLHiTEzTNdHl+gZm08evAKB/iMU6sB6kWgSH/Gt2XOygu9HbJI+zFJFLrIMHJbFb4nGGlNxbUy3X/1Gq5RqJ5HFx9LHsH27DIT5ivzIlbLZZxQeqEyc5iUk+qv6rne8qlXQvysCIuHK9E2rHcEwYjJtEdrykFEV+5Zds8XagbzafRsMerkC5aMIdY+D4er/vYQFSPPT55HXZcC3RzADAxYIzcO21DllcKfgAdi6oIjluPNOCpwxb18E+tRXQ5C+wwqnPa9s1Yau6JVvZDtIWNtXgLSwe5EDQ0ETg+qsQj/oJOW9Q7dYU7kAYPT51cs7sAADyvgXzHNByIE1SAZDVwz1T32EziO/yc47KztPDOtFbz/Ak/ExEbnv8nIh1xFIpeBIi9P6SjP0Qa36drZhnjJ8ptAxTDPM4li+ckURsQAFWN/M= 2021-05-14T07:33:18.786 INFO:teuthology.kill:2021-05-14 07:33:18,784.784 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-14T07:33:18.787 INFO:teuthology.kill:2021-05-14 07:33:18,786.786 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-14T07:33:18.788 INFO:teuthology.kill:2021-05-14 07:33:18,787.787 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-14T07:33:18.789 INFO:teuthology.kill:2021-05-14 07:33:18,788.788 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-14T07:33:18.845 INFO:teuthology.kill:2021-05-14 07:33:18,845.845 INFO:teuthology.orchestra.console:Power off smithi106 2021-05-14T07:33:18.860 INFO:teuthology.kill:2021-05-14 07:33:18,860.860 INFO:teuthology.orchestra.console:Power off smithi133 2021-05-14T07:33:18.871 INFO:teuthology.kill:2021-05-14 07:33:18,870.870 INFO:teuthology.orchestra.console:Power off smithi190 2021-05-14T07:33:18.884 INFO:teuthology.kill:2021-05-14 07:33:18,884.884 INFO:teuthology.orchestra.console:Power off smithi032 2021-05-14T07:33:27.245 INFO:teuthology.kill:2021-05-14 07:33:27,245.245 INFO:teuthology.orchestra.console:Power off for smithi133 completed 2021-05-14T07:33:27.249 INFO:teuthology.kill:2021-05-14 07:33:27,249.249 INFO:teuthology.orchestra.console:Power off for smithi190 completed 2021-05-14T07:33:27.260 INFO:teuthology.kill:2021-05-14 07:33:27,259.259 INFO:teuthology.orchestra.console:Power off for smithi032 completed 2021-05-14T07:33:31.374 INFO:teuthology.kill:2021-05-14 07:33:31,373.373 INFO:teuthology.orchestra.console:Power off for smithi106 completed 2021-05-14T07:33:31.629 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2021-05-14T07:33:31.630 INFO:teuthology.task.internal:roles: ubuntu@smithi106.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2021-05-14T07:33:31.630 INFO:teuthology.task.internal:roles: ubuntu@smithi133.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2021-05-14T07:33:31.631 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2021-05-14T07:33:31.631 INFO:teuthology.misc:Compressing logs... 2021-05-14T07:33:31.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-14T07:34:06.283 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-14T07:34:06.357 INFO:teuthology.kill:No teuthology processes running 2021-05-14T07:34:08.429 INFO:teuthology.kill:Nuking machines: ['smithi032', 'smithi106', 'smithi133', 'smithi190'] 2021-05-14T07:34:09.339 INFO:teuthology.kill:2021-05-14 07:34:09,339.339 INFO:teuthology.nuke:targets: 2021-05-14T07:34:09.341 INFO:teuthology.kill: smithi032.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQsfW6AfwbWfXYpF6MuEvXVKMRmINVjwvldGS4gM8It9AuB+MeWHbNHtGAwPIDAL6nCPfimH3h/MhQCBgPvNIV6jr8qwHGLBQ8QUgYF5Xx+0A4zIqYd0UW3wngHdf8AogiPelUS8J3cdFxBW1QVVxZMzCDkpLa5mlg3FYA3EvQNqzpv1s7tSgXfQ998pFmfddEKVT/mbipz2HVCXXz5j5ljiyoRzewbpPbkaz3MD3/Olb4M5bKqo6OUFSvA3L6/HK1lgd8TasqOITH1mNxt5uUTswmrrG4Eq7+Go4txXtshC0uGimoPUZA4vT9McnvVucx8oBNSEuGp1jMOn/Vd9LY0e5Ach5AmO+tOeDYzWogzN9pkl/YahV152LGYkosa1rAhbaaVLKOjQaaC3VxY9dnNr+PlPHIS879g9QTvsg+s8BNzNHzpXhIZlDJtpGPjvaoirfVqtQ7owCu7+01pWjdKRmGGXkwPj3jsTJlj9iYgx1kuXp/An0dpvgvQXAgqu0= 2021-05-14T07:34:09.341 INFO:teuthology.kill: smithi106.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8f8K7QRcYFmnSWc8p8HXZ9FdlUnv/Rm35q7YJS1YsQ/UKM6veoVPf3TUlEdHYnulVoodG33zv1tYrQzbj7vcV2FZ4K3JBU0gVEnD1JBokvZcYGeW1Xj3Qx2xoNE0oJ0MyUTQ+aY96EEYS9szVx2cTch/BeE4ZWQXGKMLWIjtMM5r8G8x889pCWc1NGCjpGvCIlfTW3tsmEOAL5zOoa1lumKVY7FYbJJQAgZPr+vim+M3GDRTkyt5d2Qj6rfEx2bv+VGZX+65LRj8VYOgpOCQkoCOySRyirAFaJzrooCILvTBSrB3pcVPsiSdBCIRw+0vLS6gtqxNbF5dUAwgdQyjW3DPiHisjcWz2lvirWXnNPhpYHnzMB/zUbDs+aMvpD+c82S7mtc2vSd5fvWIXktjfCBUeaB4XRQevnFRD8alHqRPAK3mM2oVeZblnszKGq23gFNRyaPxwjq6EuGQPnaueN2kmf/jWEn/nyIZ3NJH5EGB/QaS8oaP2qlimFqb4Wnc= 2021-05-14T07:34:09.342 INFO:teuthology.kill: smithi133.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvr1uYsE05Mo0/5zMMQJ9Lp+fDpWU1HyBV6x4HisXPRETIgH6/Tp7z7bd/8RaIMEVAY+NRXrBn7tj1u0TD36949S0bXA/3mRR4epT0nWOiksfYYTzmF66g9Qscz/+O5wv8z2G8kAn015F0FKGHnX2UzBYcwDdFrI7cqgrhTQg1TOGmrbwFjarY0H4S6zXVLQ/kkYymJ2ncf/Rh6Pfzm1ZROWbA+kyhBY07O+hABakPMsvVLhfE2usFMVpfYdwUiPeoDc/nX7AmcwBsEQ+TvkwDCi+8WCqzEAn5dGhNmt4R+DN4ujHvqQi3ulQJojGo2wXE9R+keqo/PL1wVb8nwACouyyT/ts45YEJR86PZLkNc3ojzjDuTFCV6VngE5b2ApsmTmJONBQh46Pn3HMr6kb0dr8X1OWWFN1vRlGn0K7JqU/MvCzeljk2rjUVTLhI2LVMYFohNMANdOWrNI7I/ZAnwfMVMIDOoNNB8o36+WZyHQmdcPg4MU7rviBzfnEBZSU= 2021-05-14T07:34:09.342 INFO:teuthology.kill: smithi190.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCHJKMm12aMLp+Gjd3zicYfa6EDvcY8Y0OJbw0pqKP0JRsiWNHiGCCnEVR5QgRQsEJ7NStLEFSjtQVzknoxLHiTEzTNdHl+gZm08evAKB/iMU6sB6kWgSH/Gt2XOygu9HbJI+zFJFLrIMHJbFb4nGGlNxbUy3X/1Gq5RqJ5HFx9LHsH27DIT5ivzIlbLZZxQeqEyc5iUk+qv6rne8qlXQvysCIuHK9E2rHcEwYjJtEdrykFEV+5Zds8XagbzafRsMerkC5aMIdY+D4er/vYQFSPPT55HXZcC3RzADAxYIzcO21DllcKfgAdi6oIjluPNOCpwxb18E+tRXQ5C+wwqnPa9s1Yau6JVvZDtIWNtXgLSwe5EDQ0ETg+qsQj/oJOW9Q7dYU7kAYPT51cs7sAADyvgXzHNByIE1SAZDVwz1T32EziO/yc47KztPDOtFbz/Ak/ExEbnv8nIh1xFIpeBIi9P6SjP0Qa36drZhnjJ8ptAxTDPM4li+ckURsQAFWN/M= 2021-05-14T07:34:09.363 INFO:teuthology.kill:2021-05-14 07:34:09,363.363 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-14T07:34:09.367 INFO:teuthology.kill:2021-05-14 07:34:09,367.367 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-14T07:34:09.368 INFO:teuthology.kill:2021-05-14 07:34:09,368.368 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-14T07:34:09.369 INFO:teuthology.kill:2021-05-14 07:34:09,369.369 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-14T07:34:09.420 INFO:teuthology.kill:2021-05-14 07:34:09,420.420 INFO:teuthology.orchestra.console:Power off smithi133 2021-05-14T07:34:09.435 INFO:teuthology.kill:2021-05-14 07:34:09,435.435 INFO:teuthology.orchestra.console:Power off smithi190 2021-05-14T07:34:09.447 INFO:teuthology.kill:2021-05-14 07:34:09,447.447 INFO:teuthology.orchestra.console:Power off smithi032 2021-05-14T07:34:09.456 INFO:teuthology.kill:2021-05-14 07:34:09,456.456 INFO:teuthology.orchestra.console:Power off smithi106 2021-05-14T07:34:09.592 INFO:teuthology.kill:2021-05-14 07:34:09,591.591 INFO:teuthology.orchestra.console:Power off for smithi190 completed 2021-05-14T07:34:09.593 INFO:teuthology.kill:2021-05-14 07:34:09,593.593 INFO:teuthology.orchestra.console:Power off for smithi133 completed 2021-05-14T07:34:09.600 INFO:teuthology.kill:2021-05-14 07:34:09,600.600 INFO:teuthology.orchestra.console:Power off for smithi032 completed 2021-05-14T07:34:09.605 INFO:teuthology.kill:2021-05-14 07:34:09,605.605 INFO:teuthology.orchestra.console:Power off for smithi106 completed 2021-05-14T07:34:09.743 INFO:teuthology.kill:2021-05-14 07:34:09,742.742 INFO:teuthology.lock.ops:unlocked smithi032.front.sepia.ceph.com 2021-05-14T07:34:09.747 INFO:teuthology.kill:2021-05-14 07:34:09,746.746 INFO:teuthology.lock.ops:unlocked smithi106.front.sepia.ceph.com 2021-05-14T07:34:09.747 INFO:teuthology.kill:2021-05-14 07:34:09,747.747 INFO:teuthology.lock.ops:unlocked smithi190.front.sepia.ceph.com 2021-05-14T07:34:09.748 INFO:teuthology.kill:2021-05-14 07:34:09,748.748 INFO:teuthology.lock.ops:unlocked smithi133.front.sepia.ceph.com 2021-05-14T07:36:09.925 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-05-14T07:36:10.052 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-05-14T07:36:10.053 INFO:teuthology.nuke:Checking targets against current locks 2021-05-14T07:36:10.128 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/scripts/dispatcher.py", line 33, in main teuthology.dispatcher.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 59, in main verbose File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 152, in run_job unlock_targets(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 196, in unlock_targets nuke(fake_ctx, True) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable