2021-05-16T03:33:14.522 INFO:root:teuthology version: 1.1.0-cc3bac84 2021-05-16T03:33:14.607 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi031.front.sepia.ceph.com' reimaging 2021-05-16T03:33:14.608 INFO:teuthology.lock.ops:Updating [ubuntu@smithi031.front.sepia.ceph.com]: reset os type and version on server 2021-05-16T03:33:14.609 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2021-05-16T03:33:14.624 INFO:teuthology.lock.ops:Node 'ubuntu@smithi031.front.sepia.ceph.com' reimaging is complete 2021-05-16T03:33:14.624 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi161.front.sepia.ceph.com' reimaging 2021-05-16T03:33:14.625 INFO:teuthology.lock.ops:Updating [ubuntu@smithi161.front.sepia.ceph.com]: reset os type and version on server 2021-05-16T03:33:14.625 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2021-05-16T03:33:14.640 INFO:teuthology.lock.ops:Node 'ubuntu@smithi161.front.sepia.ceph.com' reimaging is complete 2021-05-16T03:33:14.882 INFO:teuthology.provision.fog.smithi031:Scheduling deploy of centos 8.2 2021-05-16T03:33:14.888 INFO:teuthology.provision.fog.smithi161:Scheduling deploy of centos 8.2 2021-05-16T03:33:16.048 INFO:teuthology.orchestra.console:Power off smithi031 2021-05-16T03:33:16.149 INFO:teuthology.orchestra.console:Power off smithi161 2021-05-16T03:33:28.529 INFO:teuthology.orchestra.console:Power off for smithi031 completed 2021-05-16T03:33:28.619 INFO:teuthology.orchestra.console:Power off for smithi161 completed 2021-05-16T03:33:28.631 INFO:teuthology.orchestra.console:Power on smithi031 2021-05-16T03:33:28.721 INFO:teuthology.orchestra.console:Power on smithi161 2021-05-16T03:33:32.887 INFO:teuthology.orchestra.console:Power on for smithi031 completed 2021-05-16T03:33:32.988 INFO:teuthology.orchestra.console:Power on for smithi161 completed 2021-05-16T03:33:32.988 INFO:teuthology.provision.fog.smithi031:Waiting for deploy to finish 2021-05-16T03:33:33.089 INFO:teuthology.provision.fog.smithi161:Waiting for deploy to finish 2021-05-16T03:40:55.106 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-16T03:40:55.410 INFO:teuthology.provision.fog.smithi031:Node is ready 2021-05-16T03:40:55.427 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031.front.sepia.ceph.com 2021-05-16T03:40:55.483 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31 smithi031.front.sepia.ceph.com smithi031 2021-05-16T03:40:55.827 INFO:teuthology.provision.fog.smithi031:Deploy complete! 2021-05-16T03:42:15.945 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-16T03:42:16.226 INFO:teuthology.provision.fog.smithi161:Node is ready 2021-05-16T03:42:16.281 INFO:teuthology.orchestra.run.smithi161.stdout:smithi161.front.sepia.ceph.com 2021-05-16T03:42:16.337 INFO:teuthology.orchestra.run.smithi161.stdout:172.21.15.161 smithi161.front.sepia.ceph.com smithi161 2021-05-16T03:42:16.662 INFO:teuthology.provision.fog.smithi161:Deploy complete! 2021-05-16T03:42:16.836 INFO:teuthology.lock.ops:Checking smithi031.front.sepia.ceph.com 2021-05-16T03:42:16.837 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-16T03:42:16.865 INFO:teuthology.lock.ops:Checking smithi161.front.sepia.ceph.com 2021-05-16T03:42:16.866 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-16T03:42:16.882 INFO:teuthology.lock.ops:Updating [smithi031.front.sepia.ceph.com]: set os type and version on server 2021-05-16T03:42:16.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-16T03:42:17.170 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2021-05-16T03:42:17.227 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Linux" 2021-05-16T03:42:17.228 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="8 (Core)" 2021-05-16T03:42:17.228 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2021-05-16T03:42:17.228 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2021-05-16T03:42:17.229 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="8" 2021-05-16T03:42:17.229 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el8" 2021-05-16T03:42:17.229 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-05-16T03:42:17.229 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2021-05-16T03:42:17.230 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-16T03:42:17.230 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://www.centos.org/" 2021-05-16T03:42:17.230 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-16T03:42:17.230 INFO:teuthology.orchestra.run.smithi031.stdout: 2021-05-16T03:42:17.231 INFO:teuthology.orchestra.run.smithi031.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-16T03:42:17.231 INFO:teuthology.orchestra.run.smithi031.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-16T03:42:17.231 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-05-16T03:42:17.231 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-05-16T03:42:17.232 INFO:teuthology.orchestra.run.smithi031.stdout: 2021-05-16T03:42:17.233 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2021-05-16T03:42:17.249 INFO:teuthology.lock.ops:Updating [smithi161.front.sepia.ceph.com]: set os type and version on server 2021-05-16T03:42:17.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-16T03:42:17.515 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2021-05-16T03:42:17.569 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="CentOS Linux" 2021-05-16T03:42:17.606 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="8 (Core)" 2021-05-16T03:42:17.607 INFO:teuthology.orchestra.run.smithi161.stdout:ID="centos" 2021-05-16T03:42:17.607 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="rhel fedora" 2021-05-16T03:42:17.607 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="8" 2021-05-16T03:42:17.607 INFO:teuthology.orchestra.run.smithi161.stdout:PLATFORM_ID="platform:el8" 2021-05-16T03:42:17.608 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-05-16T03:42:17.608 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31" 2021-05-16T03:42:17.608 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-16T03:42:17.608 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://www.centos.org/" 2021-05-16T03:42:17.609 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-16T03:42:17.609 INFO:teuthology.orchestra.run.smithi161.stdout: 2021-05-16T03:42:17.609 INFO:teuthology.orchestra.run.smithi161.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-16T03:42:17.610 INFO:teuthology.orchestra.run.smithi161.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-16T03:42:17.610 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-05-16T03:42:17.610 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-05-16T03:42:17.610 INFO:teuthology.orchestra.run.smithi161.stdout: 2021-05-16T03:42:17.611 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2021-05-16T03:42:17.703 INFO:teuthology.dispatcher.supervisor:Running job 6116476 2021-05-16T03:42:17.703 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_cc3bac844fd368a871b19f94cc5d520695b86d44/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-05-16_03:31:02-rados-pacific-distro-basic-smithi/6116476 --name teuthology-2021-05-16_03:31:02-rados-pacific-distro-basic-smithi --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados tasks/rados_api_tests validater/valgrind} -- /home/teuthworker/archive/teuthology-2021-05-16_03:31:02-rados-pacific-distro-basic-smithi/6116476/orig.config.yaml 2021-05-16T03:42:17.708 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-05-16_03:31:02-rados-pacific-distro-basic-smithi/6116476 2021-05-16T03:42:17.709 INFO:teuthology.dispatcher.supervisor:Job PID: 9411 2021-05-16T03:42:17.709 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-16T15:43:02.851 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-16T15:43:03.562 INFO:teuthology.kill:Killing Pids: {9411} 2021-05-16T15:43:04.545 INFO:teuthology.kill:Nuking machines: ['smithi031', 'smithi161'] 2021-05-16T15:43:05.419 INFO:teuthology.kill:2021-05-16 15:43:05,418.418 INFO:teuthology.nuke:targets: 2021-05-16T15:43:05.420 INFO:teuthology.kill: smithi031.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDty2nPM3nmyKyCft9zckgqWqxbAYM4fAT27Tfmd3LlqwCFb76uDHokgucr0wZBfnQ3hyidP+M2hRHEZFpdwXm1IUAm+WA8X5S3W/y8o7sB94S1w4ucke4IIGJM1ltWccpHOYt8LBU5mWqAEsGFA7svu2+ndJ/1KxXINJDulZStnZpbrucV3f5PYpFiEomykefkpb86264kAuJs7zLAnv/sYgSJasZzHu+UHbz7NvbA0RL4ICun39aeHbfxG3d8q72k+OIYTg0W+qL8EK3ldL6PaTQQkUHqsNdO4bJ/yTsG3IhupAm6dqsbw1oCOboKl0t/kTUWuvvWYi8uspqesjHVmHCuiwFs2k7E/SxOzSRzAZR+e4Y+Wtg1Nz9Z0OtnLWB+a+a9c2Cry+b9y+Z6dKkbSciHkJ1gclqpmuUR9eDkn+0MBFMA0P8xsLO0BWh5qRPvCuNe1O+NPyGtGXDzGdUL6jXRl89dmXqO/ueCx2aj8kpZBv6Y1GOTQAq0NjdRybc= 2021-05-16T15:43:05.420 INFO:teuthology.kill: smithi161.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0ftlmQeERH2Mhp3H5bPtR8SXPYdvo7xvKEF6dg5iA603d/WTmag3vpk0VPt8RA2+kzRNKv6kA9tvjYVEN+9uRk3WiUsBcuQvFd+i/srcZ5KHLH6XsNDiIKbfqhLNzdB0QBrWetOi4zfHMwX7XvQ4cqihGoUojbIVAxRYN+U0VzQdBGvmF4arJnh/Y9CvlQ3j/47iFKhurvnnv0FaI5HSM2iLpRBOUylyPAfzmOz+T7JjpTfFfLeEcTbSyG5BzTXP7Iw2kHsEOlOCRQHN4B0cyS9wRU5Rt1o/3wZHTBLssjO1PNpNo9L1TkcXTLbguLYgVDs/UADfaW7lW45CmtZULV8yLi6jz0auOzJqRwGKH5IzzY3383e1/7eiBsmSUnK6SshoXfNaTcLvTNG9XArUilVJZvVIJC7Kk19IblCfzqNym0yvjdu3gaAnQ9UyRjpxdAJpywcGSbprCW7pU6kh4eQyYM3MxGFuFVrlCr2kQc12HWYfZmOAWWBP3xW7ApP0= 2021-05-16T15:43:05.420 INFO:teuthology.kill:2021-05-16 15:43:05,419.419 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-16T15:43:05.421 INFO:teuthology.kill:2021-05-16 15:43:05,421.421 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-16T15:43:05.480 INFO:teuthology.kill:2021-05-16 15:43:05,479.479 INFO:teuthology.orchestra.console:Power off smithi161 2021-05-16T15:43:05.492 INFO:teuthology.kill:2021-05-16 15:43:05,491.491 INFO:teuthology.orchestra.console:Power off smithi031 2021-05-16T15:43:13.847 INFO:teuthology.kill:2021-05-16 15:43:13,847.847 INFO:teuthology.orchestra.console:Power off for smithi161 completed 2021-05-16T15:43:13.853 INFO:teuthology.kill:2021-05-16 15:43:13,852.852 INFO:teuthology.orchestra.console:Power off for smithi031 completed 2021-05-16T15:43:14.061 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-05-16T15:43:14.061 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-05-16T15:43:14.062 INFO:teuthology.misc:Compressing logs... 2021-05-16T15:43:14.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-16T15:43:48.683 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_cc3bac844fd368a871b19f94cc5d520695b86d44/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_cc3bac844fd368a871b19f94cc5d520695b86d44/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_cc3bac844fd368a871b19f94cc5d520695b86d44/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_cc3bac844fd368a871b19f94cc5d520695b86d44/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_cc3bac844fd368a871b19f94cc5d520695b86d44/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_cc3bac844fd368a871b19f94cc5d520695b86d44/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_cc3bac844fd368a871b19f94cc5d520695b86d44/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-16T15:43:48.732 INFO:teuthology.kill:No teuthology processes running 2021-05-16T15:43:49.726 INFO:teuthology.kill:Nuking machines: ['smithi031', 'smithi161'] 2021-05-16T15:43:50.611 INFO:teuthology.kill:2021-05-16 15:43:50,611.611 INFO:teuthology.nuke:targets: 2021-05-16T15:43:50.612 INFO:teuthology.kill: smithi031.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDty2nPM3nmyKyCft9zckgqWqxbAYM4fAT27Tfmd3LlqwCFb76uDHokgucr0wZBfnQ3hyidP+M2hRHEZFpdwXm1IUAm+WA8X5S3W/y8o7sB94S1w4ucke4IIGJM1ltWccpHOYt8LBU5mWqAEsGFA7svu2+ndJ/1KxXINJDulZStnZpbrucV3f5PYpFiEomykefkpb86264kAuJs7zLAnv/sYgSJasZzHu+UHbz7NvbA0RL4ICun39aeHbfxG3d8q72k+OIYTg0W+qL8EK3ldL6PaTQQkUHqsNdO4bJ/yTsG3IhupAm6dqsbw1oCOboKl0t/kTUWuvvWYi8uspqesjHVmHCuiwFs2k7E/SxOzSRzAZR+e4Y+Wtg1Nz9Z0OtnLWB+a+a9c2Cry+b9y+Z6dKkbSciHkJ1gclqpmuUR9eDkn+0MBFMA0P8xsLO0BWh5qRPvCuNe1O+NPyGtGXDzGdUL6jXRl89dmXqO/ueCx2aj8kpZBv6Y1GOTQAq0NjdRybc= 2021-05-16T15:43:50.612 INFO:teuthology.kill: smithi161.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0ftlmQeERH2Mhp3H5bPtR8SXPYdvo7xvKEF6dg5iA603d/WTmag3vpk0VPt8RA2+kzRNKv6kA9tvjYVEN+9uRk3WiUsBcuQvFd+i/srcZ5KHLH6XsNDiIKbfqhLNzdB0QBrWetOi4zfHMwX7XvQ4cqihGoUojbIVAxRYN+U0VzQdBGvmF4arJnh/Y9CvlQ3j/47iFKhurvnnv0FaI5HSM2iLpRBOUylyPAfzmOz+T7JjpTfFfLeEcTbSyG5BzTXP7Iw2kHsEOlOCRQHN4B0cyS9wRU5Rt1o/3wZHTBLssjO1PNpNo9L1TkcXTLbguLYgVDs/UADfaW7lW45CmtZULV8yLi6jz0auOzJqRwGKH5IzzY3383e1/7eiBsmSUnK6SshoXfNaTcLvTNG9XArUilVJZvVIJC7Kk19IblCfzqNym0yvjdu3gaAnQ9UyRjpxdAJpywcGSbprCW7pU6kh4eQyYM3MxGFuFVrlCr2kQc12HWYfZmOAWWBP3xW7ApP0= 2021-05-16T15:43:50.630 INFO:teuthology.kill:2021-05-16 15:43:50,630.630 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-16T15:43:50.633 INFO:teuthology.kill:2021-05-16 15:43:50,633.633 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-16T15:43:50.688 INFO:teuthology.kill:2021-05-16 15:43:50,687.687 INFO:teuthology.orchestra.console:Power off smithi161 2021-05-16T15:43:50.700 INFO:teuthology.kill:2021-05-16 15:43:50,700.700 INFO:teuthology.orchestra.console:Power off smithi031 2021-05-16T15:43:50.827 INFO:teuthology.kill:2021-05-16 15:43:50,826.826 INFO:teuthology.orchestra.console:Power off for smithi161 completed 2021-05-16T15:43:50.840 INFO:teuthology.kill:2021-05-16 15:43:50,840.840 INFO:teuthology.orchestra.console:Power off for smithi031 completed 2021-05-16T15:43:50.965 INFO:teuthology.kill:2021-05-16 15:43:50,964.964 INFO:teuthology.lock.ops:unlocked smithi161.front.sepia.ceph.com 2021-05-16T15:43:50.975 INFO:teuthology.kill:2021-05-16 15:43:50,975.975 INFO:teuthology.lock.ops:unlocked smithi031.front.sepia.ceph.com 2021-05-16T15:45:51.232 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15