2021-05-08T18:37:15.228 INFO:root:teuthology version: 1.1.0-19220a3b 2021-05-08T18:37:15.329 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi187.front.sepia.ceph.com' reimaging 2021-05-08T18:37:15.330 INFO:teuthology.lock.ops:Updating [ubuntu@smithi187.front.sepia.ceph.com]: reset os type and version on server 2021-05-08T18:37:15.331 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2021-05-08T18:37:15.348 INFO:teuthology.lock.ops:Node 'ubuntu@smithi187.front.sepia.ceph.com' reimaging is complete 2021-05-08T18:37:15.349 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi199.front.sepia.ceph.com' reimaging 2021-05-08T18:37:15.349 INFO:teuthology.lock.ops:Updating [ubuntu@smithi199.front.sepia.ceph.com]: reset os type and version on server 2021-05-08T18:37:15.349 INFO:teuthology.lock.ops:Updating smithi199.front.sepia.ceph.com on lock server 2021-05-08T18:37:15.364 INFO:teuthology.lock.ops:Node 'ubuntu@smithi199.front.sepia.ceph.com' reimaging is complete 2021-05-08T18:37:15.562 INFO:teuthology.provision.fog.smithi187:Scheduling deploy of centos 8.2 2021-05-08T18:37:15.566 INFO:teuthology.provision.fog.smithi199:Scheduling deploy of centos 8.2 2021-05-08T18:37:15.924 INFO:teuthology.orchestra.console:Power off smithi187 2021-05-08T18:37:15.961 INFO:teuthology.orchestra.console:Power off smithi199 2021-05-08T18:37:24.300 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2021-05-08T18:37:24.327 INFO:teuthology.orchestra.console:Power off for smithi199 completed 2021-05-08T18:37:24.401 INFO:teuthology.orchestra.console:Power on smithi187 2021-05-08T18:37:24.434 INFO:teuthology.orchestra.console:Power on smithi199 2021-05-08T18:37:32.778 INFO:teuthology.orchestra.console:Power on for smithi187 completed 2021-05-08T18:37:32.802 INFO:teuthology.orchestra.console:Power on for smithi199 completed 2021-05-08T18:37:32.880 INFO:teuthology.provision.fog.smithi187:Waiting for deploy to finish 2021-05-08T18:37:32.904 INFO:teuthology.provision.fog.smithi199:Waiting for deploy to finish 2021-05-08T18:41:39.429 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-08T18:41:55.739 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-08T18:41:56.053 INFO:teuthology.provision.fog.smithi199:Node is ready 2021-05-08T18:41:56.068 INFO:teuthology.orchestra.run.smithi199.stdout:smithi199.front.sepia.ceph.com 2021-05-08T18:41:56.121 INFO:teuthology.orchestra.run.smithi199.stdout:172.21.15.199 smithi199.front.sepia.ceph.com smithi199 2021-05-08T18:41:56.442 INFO:teuthology.provision.fog.smithi199:Deploy complete! 2021-05-08T18:42:00.230 INFO:teuthology.provision.fog.smithi187:Node is ready 2021-05-08T18:42:00.249 INFO:teuthology.orchestra.run.smithi187.stdout:smithi187.front.sepia.ceph.com 2021-05-08T18:42:00.305 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.15.187 smithi187.front.sepia.ceph.com smithi187 2021-05-08T18:42:00.656 INFO:teuthology.provision.fog.smithi187:Deploy complete! 2021-05-08T18:42:00.775 INFO:teuthology.lock.ops:Checking smithi187.front.sepia.ceph.com 2021-05-08T18:42:00.775 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-08T18:42:00.791 INFO:teuthology.lock.ops:Checking smithi199.front.sepia.ceph.com 2021-05-08T18:42:00.791 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-08T18:42:00.805 INFO:teuthology.lock.ops:Updating [smithi187.front.sepia.ceph.com]: set os type and version on server 2021-05-08T18:42:00.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-08T18:42:01.071 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2021-05-08T18:42:01.125 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Linux" 2021-05-08T18:42:01.125 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="8 (Core)" 2021-05-08T18:42:01.125 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos" 2021-05-08T18:42:01.126 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora" 2021-05-08T18:42:01.126 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="8" 2021-05-08T18:42:01.126 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el8" 2021-05-08T18:42:01.126 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-05-08T18:42:01.127 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2021-05-08T18:42:01.127 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-08T18:42:01.127 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://www.centos.org/" 2021-05-08T18:42:01.128 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-08T18:42:01.128 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-05-08T18:42:01.128 INFO:teuthology.orchestra.run.smithi187.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-08T18:42:01.129 INFO:teuthology.orchestra.run.smithi187.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-08T18:42:01.129 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-05-08T18:42:01.130 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-05-08T18:42:01.130 INFO:teuthology.orchestra.run.smithi187.stdout: 2021-05-08T18:42:01.131 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2021-05-08T18:42:01.147 INFO:teuthology.lock.ops:Updating [smithi199.front.sepia.ceph.com]: set os type and version on server 2021-05-08T18:42:01.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-08T18:42:01.424 INFO:teuthology.orchestra.run.smithi199.stdout:x86_64 2021-05-08T18:42:01.478 INFO:teuthology.orchestra.run.smithi199.stdout:NAME="CentOS Linux" 2021-05-08T18:42:01.478 INFO:teuthology.orchestra.run.smithi199.stdout:VERSION="8 (Core)" 2021-05-08T18:42:01.479 INFO:teuthology.orchestra.run.smithi199.stdout:ID="centos" 2021-05-08T18:42:01.479 INFO:teuthology.orchestra.run.smithi199.stdout:ID_LIKE="rhel fedora" 2021-05-08T18:42:01.479 INFO:teuthology.orchestra.run.smithi199.stdout:VERSION_ID="8" 2021-05-08T18:42:01.479 INFO:teuthology.orchestra.run.smithi199.stdout:PLATFORM_ID="platform:el8" 2021-05-08T18:42:01.479 INFO:teuthology.orchestra.run.smithi199.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-05-08T18:42:01.480 INFO:teuthology.orchestra.run.smithi199.stdout:ANSI_COLOR="0;31" 2021-05-08T18:42:01.480 INFO:teuthology.orchestra.run.smithi199.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-08T18:42:01.480 INFO:teuthology.orchestra.run.smithi199.stdout:HOME_URL="https://www.centos.org/" 2021-05-08T18:42:01.480 INFO:teuthology.orchestra.run.smithi199.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-08T18:42:01.481 INFO:teuthology.orchestra.run.smithi199.stdout: 2021-05-08T18:42:01.481 INFO:teuthology.orchestra.run.smithi199.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-08T18:42:01.481 INFO:teuthology.orchestra.run.smithi199.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-08T18:42:01.481 INFO:teuthology.orchestra.run.smithi199.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-05-08T18:42:01.481 INFO:teuthology.orchestra.run.smithi199.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-05-08T18:42:01.482 INFO:teuthology.orchestra.run.smithi199.stdout: 2021-05-08T18:42:01.482 INFO:teuthology.lock.ops:Updating smithi199.front.sepia.ceph.com on lock server 2021-05-08T18:42:01.564 INFO:teuthology.dispatcher.supervisor:Running job 6103734 2021-05-08T18:42:01.564 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-05-07_03:31:03-rados-pacific-distro-basic-smithi/6103734 --name teuthology-2021-05-07_03:31:03-rados-pacific-distro-basic-smithi --description rados/cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} -- /home/teuthworker/archive/teuthology-2021-05-07_03:31:03-rados-pacific-distro-basic-smithi/6103734/orig.config.yaml 2021-05-08T18:42:01.567 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-05-07_03:31:03-rados-pacific-distro-basic-smithi/6103734 2021-05-08T18:42:01.568 INFO:teuthology.dispatcher.supervisor:Job PID: 17917 2021-05-08T18:42:01.568 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-09T06:42:51.227 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-09T06:42:51.520 INFO:teuthology.kill:Killing Pids: {17917} 2021-05-09T06:42:52.652 INFO:teuthology.kill:Nuking machines: ['smithi187', 'smithi199'] 2021-05-09T06:42:53.628 INFO:teuthology.kill:2021-05-09 06:42:53,627.627 INFO:teuthology.nuke:targets: 2021-05-09T06:42:53.629 INFO:teuthology.kill: smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDN7mqOmsHWdC0Ad3l9iBuruWXL0eRd/rMptAXWlWg8I72u/7dYePMn12FloZ/ReQwYtZh8Sl+6s8slMQLpveaLHWDtAagHpiynzQj5JNEwLJ/20XOckBCQHx/SnLfnWCEIhvg5I4yJe1A3GfnYoAnigLBqW5QPyhx3cvNqtWYNvjxKSJMhXugaCnCcqoYdUCbWBiNyv8Fs0i4iq6V8p/9e3vQn4cLQQcWlUKWH7Y62Q7jHrvybug1Ramqk4mwlRjDrGEHONaVXT46yqaAi4CvQ136MVLRd8yEXw6oK27Sew3hR8/7xcHjgdT12dBySeK8O549X+1lAcJCgOmbaDFN4QurbL3Hy2X+Gegp0xhrv6I2KMn8FaFqreU9bzQsc1O6pRaJm3zR+d016K4NSIcOSr60fHb8aUC10X/CojwVz7e56fBW7b20+nVsT+Y3M1ThwjtHPNkc7DzJqYlWaUGVTsSWuHXIHsMzjCX5EqzD6pdJ8XrFsotkPKyYm2aMfBuU= 2021-05-09T06:42:53.629 INFO:teuthology.kill: smithi199.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmI1M2rzglnorK6p1PIZjr7HpuVLRlJdPkmvASGh2ha+cNQHg2yn+OfwTF6g7geqWhmHNCXB4lK72KZu0fmGH6BRkA2+m5pekzxQhWQqWflzhFmGiwe546Day4PZEOMcNscdQzJ8TsFE+9ZjpwyvXQWDtCuJxBCEk9Rob4lU9f/CLYHm+7gPOF8cdBypaxoTW4HNQSY1ykIC8WUQJsfRgj0+oiTvkrRcyv8kC2MX3660xoMsDnQEwhPrqXWAQcQPkF/jiVLfv/1wyw+DlP60osDL7sxv+c22hdl1mzi+Nna7JJS6y8C0KDzDZmJWTN6CDf6oQ94S7tG/QLkZahhtAemo+GMhybTCP0jf04J/V7w5yXj1sAEz+LZScEC8XUXLgou5Qu2rHr/t8Cv3W9+U90eZZc2WjzDdDpFVZk82+Vx4K63TX/M/lJbfyRu7TFjC8ZIXz3Ei0uD3yL7xF+yxV1/Rw3Onbqfb04Uts1C1n6Ar7xbcP7HuOOw+FrruhMa5c= 2021-05-09T06:42:53.630 INFO:teuthology.kill:2021-05-09 06:42:53,628.628 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-09T06:42:53.630 INFO:teuthology.kill:2021-05-09 06:42:53,630.630 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-09T06:42:53.684 INFO:teuthology.kill:2021-05-09 06:42:53,683.683 INFO:teuthology.orchestra.console:Power off smithi187 2021-05-09T06:42:53.699 INFO:teuthology.kill:2021-05-09 06:42:53,698.698 INFO:teuthology.orchestra.console:Power off smithi199 2021-05-09T06:43:02.067 INFO:teuthology.kill:2021-05-09 06:43:02,067.067 INFO:teuthology.orchestra.console:Power off for smithi199 completed 2021-05-09T06:43:06.173 INFO:teuthology.kill:2021-05-09 06:43:06,173.173 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2021-05-09T06:43:06.422 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2021-05-09T06:43:06.423 INFO:teuthology.task.internal:roles: ubuntu@smithi199.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2021-05-09T06:43:06.423 INFO:teuthology.misc:Compressing logs... 2021-05-09T06:43:06.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-09T06:43:41.020 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-09T06:43:41.080 INFO:teuthology.kill:No teuthology processes running 2021-05-09T06:43:42.640 INFO:teuthology.kill:Nuking machines: ['smithi187', 'smithi199'] 2021-05-09T06:43:43.751 INFO:teuthology.kill:2021-05-09 06:43:43,750.750 INFO:teuthology.nuke:targets: 2021-05-09T06:43:43.752 INFO:teuthology.kill: smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDN7mqOmsHWdC0Ad3l9iBuruWXL0eRd/rMptAXWlWg8I72u/7dYePMn12FloZ/ReQwYtZh8Sl+6s8slMQLpveaLHWDtAagHpiynzQj5JNEwLJ/20XOckBCQHx/SnLfnWCEIhvg5I4yJe1A3GfnYoAnigLBqW5QPyhx3cvNqtWYNvjxKSJMhXugaCnCcqoYdUCbWBiNyv8Fs0i4iq6V8p/9e3vQn4cLQQcWlUKWH7Y62Q7jHrvybug1Ramqk4mwlRjDrGEHONaVXT46yqaAi4CvQ136MVLRd8yEXw6oK27Sew3hR8/7xcHjgdT12dBySeK8O549X+1lAcJCgOmbaDFN4QurbL3Hy2X+Gegp0xhrv6I2KMn8FaFqreU9bzQsc1O6pRaJm3zR+d016K4NSIcOSr60fHb8aUC10X/CojwVz7e56fBW7b20+nVsT+Y3M1ThwjtHPNkc7DzJqYlWaUGVTsSWuHXIHsMzjCX5EqzD6pdJ8XrFsotkPKyYm2aMfBuU= 2021-05-09T06:43:43.753 INFO:teuthology.kill: smithi199.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmI1M2rzglnorK6p1PIZjr7HpuVLRlJdPkmvASGh2ha+cNQHg2yn+OfwTF6g7geqWhmHNCXB4lK72KZu0fmGH6BRkA2+m5pekzxQhWQqWflzhFmGiwe546Day4PZEOMcNscdQzJ8TsFE+9ZjpwyvXQWDtCuJxBCEk9Rob4lU9f/CLYHm+7gPOF8cdBypaxoTW4HNQSY1ykIC8WUQJsfRgj0+oiTvkrRcyv8kC2MX3660xoMsDnQEwhPrqXWAQcQPkF/jiVLfv/1wyw+DlP60osDL7sxv+c22hdl1mzi+Nna7JJS6y8C0KDzDZmJWTN6CDf6oQ94S7tG/QLkZahhtAemo+GMhybTCP0jf04J/V7w5yXj1sAEz+LZScEC8XUXLgou5Qu2rHr/t8Cv3W9+U90eZZc2WjzDdDpFVZk82+Vx4K63TX/M/lJbfyRu7TFjC8ZIXz3Ei0uD3yL7xF+yxV1/Rw3Onbqfb04Uts1C1n6Ar7xbcP7HuOOw+FrruhMa5c= 2021-05-09T06:43:43.774 INFO:teuthology.kill:2021-05-09 06:43:43,773.773 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-09T06:43:43.778 INFO:teuthology.kill:2021-05-09 06:43:43,777.777 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-09T06:43:43.838 INFO:teuthology.kill:2021-05-09 06:43:43,837.837 INFO:teuthology.orchestra.console:Power off smithi199 2021-05-09T06:43:43.851 INFO:teuthology.kill:2021-05-09 06:43:43,851.851 INFO:teuthology.orchestra.console:Power off smithi187 2021-05-09T06:43:43.979 INFO:teuthology.kill:2021-05-09 06:43:43,978.978 INFO:teuthology.orchestra.console:Power off for smithi199 completed 2021-05-09T06:43:43.988 INFO:teuthology.kill:2021-05-09 06:43:43,988.988 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2021-05-09T06:43:44.121 INFO:teuthology.kill:2021-05-09 06:43:44,120.120 INFO:teuthology.lock.ops:unlocked smithi199.front.sepia.ceph.com 2021-05-09T06:43:44.125 INFO:teuthology.kill:2021-05-09 06:43:44,125.125 INFO:teuthology.lock.ops:unlocked smithi187.front.sepia.ceph.com 2021-05-09T06:45:44.414 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-05-09T06:45:44.498 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-05-09T06:45:44.499 INFO:teuthology.nuke:Checking targets against current locks 2021-05-09T06:45:44.568 INFO:teuthology.nuke:Not nuking smithi187.front.sepia.ceph.com because description doesn't match 2021-05-09T06:45:44.569 INFO:teuthology.nuke:Not nuking smithi199.front.sepia.ceph.com because description doesn't match