2021-05-09T13:04:59.570 INFO:root:teuthology version: 1.1.0-19220a3b 2021-05-09T13:04:59.695 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi084.front.sepia.ceph.com' reimaging 2021-05-09T13:04:59.696 INFO:teuthology.lock.ops:Updating [ubuntu@smithi084.front.sepia.ceph.com]: reset os type and version on server 2021-05-09T13:04:59.696 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2021-05-09T13:04:59.711 INFO:teuthology.lock.ops:Node 'ubuntu@smithi084.front.sepia.ceph.com' reimaging is complete 2021-05-09T13:04:59.712 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi100.front.sepia.ceph.com' reimaging 2021-05-09T13:04:59.712 INFO:teuthology.lock.ops:Updating [ubuntu@smithi100.front.sepia.ceph.com]: reset os type and version on server 2021-05-09T13:04:59.712 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2021-05-09T13:04:59.729 INFO:teuthology.lock.ops:Node 'ubuntu@smithi100.front.sepia.ceph.com' reimaging is complete 2021-05-09T13:04:59.908 INFO:teuthology.provision.fog.smithi084:Scheduling deploy of centos 8.2 2021-05-09T13:04:59.923 INFO:teuthology.provision.fog.smithi100:Scheduling deploy of centos 8.2 2021-05-09T13:05:00.243 INFO:teuthology.orchestra.console:Power off smithi084 2021-05-09T13:05:00.270 INFO:teuthology.orchestra.console:Power off smithi100 2021-05-09T13:05:08.659 INFO:teuthology.orchestra.console:Power off for smithi100 completed 2021-05-09T13:05:08.763 INFO:teuthology.orchestra.console:Power on smithi100 2021-05-09T13:05:12.788 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2021-05-09T13:05:12.890 INFO:teuthology.orchestra.console:Power on smithi084 2021-05-09T13:05:17.147 INFO:teuthology.orchestra.console:Power on for smithi084 completed 2021-05-09T13:05:17.169 INFO:teuthology.orchestra.console:Power on for smithi100 completed 2021-05-09T13:05:17.253 INFO:teuthology.provision.fog.smithi084:Waiting for deploy to finish 2021-05-09T13:05:17.270 INFO:teuthology.provision.fog.smithi100:Waiting for deploy to finish 2021-05-09T13:09:26.123 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-09T13:09:46.848 INFO:teuthology.provision.fog.smithi100:Node is ready 2021-05-09T13:09:46.872 INFO:teuthology.orchestra.run.smithi100.stdout:smithi100.front.sepia.ceph.com 2021-05-09T13:09:46.930 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.15.100 smithi100.front.sepia.ceph.com smithi100 2021-05-09T13:09:47.285 INFO:teuthology.provision.fog.smithi100:Deploy complete! 2021-05-09T13:09:49.010 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-09T13:09:49.282 INFO:teuthology.provision.fog.smithi084:Node is ready 2021-05-09T13:09:49.297 INFO:teuthology.orchestra.run.smithi084.stdout:smithi084.front.sepia.ceph.com 2021-05-09T13:09:49.352 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84 smithi084.front.sepia.ceph.com smithi084 2021-05-09T13:09:49.673 INFO:teuthology.provision.fog.smithi084:Deploy complete! 2021-05-09T13:09:49.825 INFO:teuthology.lock.ops:Checking smithi084.front.sepia.ceph.com 2021-05-09T13:09:49.827 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-09T13:09:49.844 INFO:teuthology.lock.ops:Checking smithi100.front.sepia.ceph.com 2021-05-09T13:09:49.844 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-09T13:09:49.859 INFO:teuthology.lock.ops:Updating [smithi084.front.sepia.ceph.com]: set os type and version on server 2021-05-09T13:09:49.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-09T13:09:50.123 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2021-05-09T13:09:50.177 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="CentOS Linux" 2021-05-09T13:09:50.177 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="8 (Core)" 2021-05-09T13:09:50.177 INFO:teuthology.orchestra.run.smithi084.stdout:ID="centos" 2021-05-09T13:09:50.178 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="rhel fedora" 2021-05-09T13:09:50.178 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="8" 2021-05-09T13:09:50.178 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el8" 2021-05-09T13:09:50.178 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-05-09T13:09:50.179 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2021-05-09T13:09:50.179 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-09T13:09:50.179 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://www.centos.org/" 2021-05-09T13:09:50.179 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-09T13:09:50.180 INFO:teuthology.orchestra.run.smithi084.stdout: 2021-05-09T13:09:50.180 INFO:teuthology.orchestra.run.smithi084.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-09T13:09:50.180 INFO:teuthology.orchestra.run.smithi084.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-09T13:09:50.180 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-05-09T13:09:50.180 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-05-09T13:09:50.181 INFO:teuthology.orchestra.run.smithi084.stdout: 2021-05-09T13:09:50.182 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2021-05-09T13:09:50.198 INFO:teuthology.lock.ops:Updating [smithi100.front.sepia.ceph.com]: set os type and version on server 2021-05-09T13:09:50.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-09T13:09:50.475 INFO:teuthology.orchestra.run.smithi100.stdout:x86_64 2021-05-09T13:09:50.529 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="CentOS Linux" 2021-05-09T13:09:50.530 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="8 (Core)" 2021-05-09T13:09:50.530 INFO:teuthology.orchestra.run.smithi100.stdout:ID="centos" 2021-05-09T13:09:50.530 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE="rhel fedora" 2021-05-09T13:09:50.530 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="8" 2021-05-09T13:09:50.531 INFO:teuthology.orchestra.run.smithi100.stdout:PLATFORM_ID="platform:el8" 2021-05-09T13:09:50.531 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-05-09T13:09:50.531 INFO:teuthology.orchestra.run.smithi100.stdout:ANSI_COLOR="0;31" 2021-05-09T13:09:50.531 INFO:teuthology.orchestra.run.smithi100.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-09T13:09:50.531 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://www.centos.org/" 2021-05-09T13:09:50.532 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-09T13:09:50.532 INFO:teuthology.orchestra.run.smithi100.stdout: 2021-05-09T13:09:50.532 INFO:teuthology.orchestra.run.smithi100.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-09T13:09:50.532 INFO:teuthology.orchestra.run.smithi100.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-09T13:09:50.533 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-05-09T13:09:50.533 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-05-09T13:09:50.533 INFO:teuthology.orchestra.run.smithi100.stdout: 2021-05-09T13:09:50.534 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2021-05-09T13:09:50.610 INFO:teuthology.dispatcher.supervisor:Running job 6106231 2021-05-09T13:09:50.611 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-05-09_03:31:02-rados-pacific-distro-basic-smithi/6106231 --name teuthology-2021-05-09_03:31:02-rados-pacific-distro-basic-smithi --description rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-fast-read} -- /home/teuthworker/archive/teuthology-2021-05-09_03:31:02-rados-pacific-distro-basic-smithi/6106231/orig.config.yaml 2021-05-09T13:09:50.614 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-05-09_03:31:02-rados-pacific-distro-basic-smithi/6106231 2021-05-09T13:09:50.615 INFO:teuthology.dispatcher.supervisor:Job PID: 17341 2021-05-09T13:09:50.615 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-10T01:10:41.716 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-10T01:10:44.467 INFO:teuthology.kill:Killing Pids: {17341} 2021-05-10T01:10:46.257 INFO:teuthology.kill:Nuking machines: ['smithi084', 'smithi100'] 2021-05-10T01:10:47.199 INFO:teuthology.kill:2021-05-10 01:10:47,198.198 INFO:teuthology.nuke:targets: 2021-05-10T01:10:47.200 INFO:teuthology.kill: smithi084.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGw9AsdCzlF4CXYJQ583BF1QObEAAi+BcEYuHOrZCgEYFnYcUjClutf8J665NjmZzOm7l4lH55V9fqnrKXufBylhfrK/Oya0Wqh2geproZ1bMKDlCXwDuAYS37F+M99id93W1FlKr/U/wihYULnaqi9dH0HTqTqC7qWmBqxXb+1NGKUMobX8C88kFkGB1K/HB9B3qpSp5j2TinKADxCewe16CU2DKLZZkXTDI8GfXvPF01Ne179uR+xHCK9sJR4/2jASPHQ7lHj9JF+tRtZSE0B3Qhd/CsM6h0h0xlQfwgAncqiZZ2Guf744AReQm2nf9SEIPAGV6EJjpdApW6qHNJgIXxmyZg0xvG8i3RkZQrvXjIyeTWI+Mhnw8NLoGQ4EtQ2qZC94pkrBspvHDFNNZju31kyZ6n7WU6VP/nFTAzQa4WKnLcffQDCevyIHnAPHb4fm9DoMdV4aFnX0f5om8U5Ll7P4CPb1AD0/iJ5/ZSjbe8vc3VSbSRkTbqeUItKEc= 2021-05-10T01:10:47.200 INFO:teuthology.kill: smithi100.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCEVDeUPVJ9ZhkovrxSFHRfypgNiVnUVmxteO30gnuAPd4lQwvKCM2mf5IDG9Yfz+9fCVM7z+K7kUwGsxPZS5Y6so6qljhDIs6SUYgyJWh3Pc2wrRW/qK5wSkK83sV+KQy6vPesif91L+/VOsYRfuDV7i89hFa6Pcrl3iBop/J+K3aeDUxue/G/f4dqLfW19KWZBYG9mNBDeu1oLxdh+e003+/QDWxL3Fms050oNmhbaVDg+mS1XxQx810yWqgtYQuMARECzxcM/IIRdS7xXL7X9lbC8CQpAkbKJsn0jE5Eb01kjJDdPoXQM0hfca8lPpC/BQt8s3Nf+i8b+X//zdgbQyJlePEchm3faPuqX//CDhYrEpxfvowm6paJ4A/HJWia2algJQTZa8q4GyIsuZPPRUsV8XdlR3BWsOMMHMHZwTaIsRw2nczpTjKlF1NMd+B+q1WjmnOJBGDMxY1b/KJcXTJuyZmo4V2jTrPy1GwFonZ60f3e0gACZ/4ak1tK20= 2021-05-10T01:10:47.200 INFO:teuthology.kill:2021-05-10 01:10:47,199.199 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-10T01:10:47.201 INFO:teuthology.kill:2021-05-10 01:10:47,201.201 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-10T01:10:47.263 INFO:teuthology.kill:2021-05-10 01:10:47,263.263 INFO:teuthology.orchestra.console:Power off smithi100 2021-05-10T01:10:47.281 INFO:teuthology.kill:2021-05-10 01:10:47,281.281 INFO:teuthology.orchestra.console:Power off smithi084 2021-05-10T01:10:55.664 INFO:teuthology.kill:2021-05-10 01:10:55,664.664 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2021-05-10T01:10:59.782 INFO:teuthology.kill:2021-05-10 01:10:59,781.781 INFO:teuthology.orchestra.console:Power off for smithi100 completed 2021-05-10T01:11:00.006 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-05-10T01:11:00.006 INFO:teuthology.task.internal:roles: ubuntu@smithi100.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-05-10T01:11:00.007 INFO:teuthology.misc:Compressing logs... 2021-05-10T01:11:00.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-10T01:11:34.492 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-10T01:11:34.548 INFO:teuthology.kill:No teuthology processes running 2021-05-10T01:11:35.629 INFO:teuthology.kill:Nuking machines: ['smithi084', 'smithi100'] 2021-05-10T01:11:36.519 INFO:teuthology.kill:2021-05-10 01:11:36,518.518 INFO:teuthology.nuke:targets: 2021-05-10T01:11:36.520 INFO:teuthology.kill: smithi084.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGw9AsdCzlF4CXYJQ583BF1QObEAAi+BcEYuHOrZCgEYFnYcUjClutf8J665NjmZzOm7l4lH55V9fqnrKXufBylhfrK/Oya0Wqh2geproZ1bMKDlCXwDuAYS37F+M99id93W1FlKr/U/wihYULnaqi9dH0HTqTqC7qWmBqxXb+1NGKUMobX8C88kFkGB1K/HB9B3qpSp5j2TinKADxCewe16CU2DKLZZkXTDI8GfXvPF01Ne179uR+xHCK9sJR4/2jASPHQ7lHj9JF+tRtZSE0B3Qhd/CsM6h0h0xlQfwgAncqiZZ2Guf744AReQm2nf9SEIPAGV6EJjpdApW6qHNJgIXxmyZg0xvG8i3RkZQrvXjIyeTWI+Mhnw8NLoGQ4EtQ2qZC94pkrBspvHDFNNZju31kyZ6n7WU6VP/nFTAzQa4WKnLcffQDCevyIHnAPHb4fm9DoMdV4aFnX0f5om8U5Ll7P4CPb1AD0/iJ5/ZSjbe8vc3VSbSRkTbqeUItKEc= 2021-05-10T01:11:36.520 INFO:teuthology.kill: smithi100.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCEVDeUPVJ9ZhkovrxSFHRfypgNiVnUVmxteO30gnuAPd4lQwvKCM2mf5IDG9Yfz+9fCVM7z+K7kUwGsxPZS5Y6so6qljhDIs6SUYgyJWh3Pc2wrRW/qK5wSkK83sV+KQy6vPesif91L+/VOsYRfuDV7i89hFa6Pcrl3iBop/J+K3aeDUxue/G/f4dqLfW19KWZBYG9mNBDeu1oLxdh+e003+/QDWxL3Fms050oNmhbaVDg+mS1XxQx810yWqgtYQuMARECzxcM/IIRdS7xXL7X9lbC8CQpAkbKJsn0jE5Eb01kjJDdPoXQM0hfca8lPpC/BQt8s3Nf+i8b+X//zdgbQyJlePEchm3faPuqX//CDhYrEpxfvowm6paJ4A/HJWia2algJQTZa8q4GyIsuZPPRUsV8XdlR3BWsOMMHMHZwTaIsRw2nczpTjKlF1NMd+B+q1WjmnOJBGDMxY1b/KJcXTJuyZmo4V2jTrPy1GwFonZ60f3e0gACZ/4ak1tK20= 2021-05-10T01:11:36.540 INFO:teuthology.kill:2021-05-10 01:11:36,540.540 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-10T01:11:36.543 INFO:teuthology.kill:2021-05-10 01:11:36,543.543 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-10T01:11:36.598 INFO:teuthology.kill:2021-05-10 01:11:36,597.597 INFO:teuthology.orchestra.console:Power off smithi084 2021-05-10T01:11:36.611 INFO:teuthology.kill:2021-05-10 01:11:36,610.610 INFO:teuthology.orchestra.console:Power off smithi100 2021-05-10T01:11:36.741 INFO:teuthology.kill:2021-05-10 01:11:36,740.740 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2021-05-10T01:11:36.752 INFO:teuthology.kill:2021-05-10 01:11:36,752.752 INFO:teuthology.orchestra.console:Power off for smithi100 completed 2021-05-10T01:11:36.883 INFO:teuthology.kill:2021-05-10 01:11:36,883.883 INFO:teuthology.lock.ops:unlocked smithi084.front.sepia.ceph.com 2021-05-10T01:11:36.890 INFO:teuthology.kill:2021-05-10 01:11:36,890.890 INFO:teuthology.lock.ops:unlocked smithi100.front.sepia.ceph.com 2021-05-10T01:13:37.163 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-05-10T01:13:37.256 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-05-10T01:13:37.257 INFO:teuthology.nuke:Checking targets against current locks 2021-05-10T01:13:37.334 INFO:teuthology.nuke:Not nuking smithi084.front.sepia.ceph.com because description doesn't match 2021-05-10T01:13:37.335 INFO:teuthology.nuke:Not nuking smithi100.front.sepia.ceph.com because description doesn't match