2021-08-12T16:58:57.520 INFO:root:teuthology version: 1.1.0-321319b1 2021-08-12T16:58:57.664 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi114.front.sepia.ceph.com' reimaging 2021-08-12T16:58:57.668 INFO:teuthology.lock.ops:Updating [ubuntu@smithi114.front.sepia.ceph.com]: reset os type and version on server 2021-08-12T16:58:57.669 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2021-08-12T16:58:57.697 INFO:teuthology.lock.ops:Node 'ubuntu@smithi114.front.sepia.ceph.com' reimaging is complete 2021-08-12T16:58:57.697 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi123.front.sepia.ceph.com' reimaging 2021-08-12T16:58:57.698 INFO:teuthology.lock.ops:Updating [ubuntu@smithi123.front.sepia.ceph.com]: reset os type and version on server 2021-08-12T16:58:57.700 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2021-08-12T16:58:57.743 INFO:teuthology.lock.ops:Node 'ubuntu@smithi123.front.sepia.ceph.com' reimaging is complete 2021-08-12T16:58:57.914 INFO:teuthology.provision.fog.smithi114:Scheduling deploy of centos 8.2 2021-08-12T16:58:57.948 INFO:teuthology.provision.fog.smithi123:Scheduling deploy of centos 8.2 2021-08-12T16:58:58.480 INFO:teuthology.orchestra.console:Power off smithi114 2021-08-12T16:58:58.629 INFO:teuthology.orchestra.console:Power off for smithi114 completed 2021-08-12T16:58:58.731 INFO:teuthology.orchestra.console:Power on smithi114 2021-08-12T16:58:58.914 INFO:teuthology.orchestra.console:Power off smithi123 2021-08-12T16:58:59.062 INFO:teuthology.orchestra.console:Power off for smithi123 completed 2021-08-12T16:58:59.167 INFO:teuthology.orchestra.console:Power on smithi123 2021-08-12T16:59:02.990 INFO:teuthology.orchestra.console:Power on for smithi114 completed 2021-08-12T16:59:03.092 INFO:teuthology.provision.fog.smithi114:Waiting for deploy to finish 2021-08-12T16:59:07.572 INFO:teuthology.orchestra.console:Power on for smithi123 completed 2021-08-12T16:59:07.674 INFO:teuthology.provision.fog.smithi123:Waiting for deploy to finish 2021-08-12T17:03:35.277 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-12T17:03:35.578 INFO:teuthology.provision.fog.smithi114:Node is ready 2021-08-12T17:03:35.634 INFO:teuthology.orchestra.run.smithi114.stdout:smithi114.front.sepia.ceph.com 2021-08-12T17:03:35.688 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114 smithi114.front.sepia.ceph.com smithi114 2021-08-12T17:03:36.015 INFO:teuthology.provision.fog.smithi114:Deploy complete! 2021-08-12T17:03:40.142 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-12T17:03:40.454 INFO:teuthology.provision.fog.smithi123:Node is ready 2021-08-12T17:03:40.474 INFO:teuthology.orchestra.run.smithi123.stdout:smithi123.front.sepia.ceph.com 2021-08-12T17:03:40.531 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123 smithi123.front.sepia.ceph.com smithi123 2021-08-12T17:03:40.858 INFO:teuthology.provision.fog.smithi123:Deploy complete! 2021-08-12T17:03:41.004 INFO:teuthology.lock.ops:Checking smithi114.front.sepia.ceph.com 2021-08-12T17:03:41.005 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-12T17:03:41.024 INFO:teuthology.lock.ops:Checking smithi123.front.sepia.ceph.com 2021-08-12T17:03:41.024 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-12T17:03:41.039 INFO:teuthology.lock.ops:Updating [smithi114.front.sepia.ceph.com]: set os type and version on server 2021-08-12T17:03:41.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-12T17:03:41.304 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2021-08-12T17:03:41.359 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Linux" 2021-08-12T17:03:41.359 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="8 (Core)" 2021-08-12T17:03:41.360 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2021-08-12T17:03:41.360 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2021-08-12T17:03:41.360 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="8" 2021-08-12T17:03:41.360 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el8" 2021-08-12T17:03:41.361 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-08-12T17:03:41.361 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2021-08-12T17:03:41.361 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-12T17:03:41.361 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://www.centos.org/" 2021-08-12T17:03:41.362 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-08-12T17:03:41.362 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-08-12T17:03:41.362 INFO:teuthology.orchestra.run.smithi114.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-08-12T17:03:41.362 INFO:teuthology.orchestra.run.smithi114.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-08-12T17:03:41.363 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-08-12T17:03:41.363 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-08-12T17:03:41.363 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-08-12T17:03:41.365 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2021-08-12T17:03:41.383 INFO:teuthology.lock.ops:Updating [smithi123.front.sepia.ceph.com]: set os type and version on server 2021-08-12T17:03:41.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-12T17:03:41.634 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2021-08-12T17:03:41.689 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="CentOS Linux" 2021-08-12T17:03:41.689 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="8 (Core)" 2021-08-12T17:03:41.689 INFO:teuthology.orchestra.run.smithi123.stdout:ID="centos" 2021-08-12T17:03:41.690 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="rhel fedora" 2021-08-12T17:03:41.690 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="8" 2021-08-12T17:03:41.690 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el8" 2021-08-12T17:03:41.690 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-08-12T17:03:41.691 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31" 2021-08-12T17:03:41.691 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-12T17:03:41.698 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://www.centos.org/" 2021-08-12T17:03:41.699 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-08-12T17:03:41.699 INFO:teuthology.orchestra.run.smithi123.stdout: 2021-08-12T17:03:41.699 INFO:teuthology.orchestra.run.smithi123.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-08-12T17:03:41.699 INFO:teuthology.orchestra.run.smithi123.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-08-12T17:03:41.700 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-08-12T17:03:41.700 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-08-12T17:03:41.700 INFO:teuthology.orchestra.run.smithi123.stdout: 2021-08-12T17:03:41.701 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2021-08-12T17:03:41.779 INFO:teuthology.dispatcher.supervisor:Running job 6335845 2021-08-12T17:03:41.779 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-08-12_15:36:23-rados-wip-yuri3-testing-2021-08-09-1006-distro-basic-smithi/6335845 --name yuriw-2021-08-12_15:36:23-rados-wip-yuri3-testing-2021-08-09-1006-distro-basic-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-centos_8.2_kubic_stable 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/yuriw-2021-08-12_15:36:23-rados-wip-yuri3-testing-2021-08-09-1006-distro-basic-smithi/6335845/orig.config.yaml 2021-08-12T17:03:41.782 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-08-12_15:36:23-rados-wip-yuri3-testing-2021-08-09-1006-distro-basic-smithi/6335845 2021-08-12T17:03:41.783 INFO:teuthology.dispatcher.supervisor:Job PID: 31577 2021-08-12T17:03:41.784 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-08-13T05:04:32.361 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-08-13T05:04:32.587 INFO:teuthology.kill:Killing Pids: {31577} 2021-08-13T05:04:33.692 INFO:teuthology.kill:Nuking machines: ['smithi114', 'smithi123'] 2021-08-13T05:04:34.681 INFO:teuthology.kill:2021-08-13 05:04:34,681.681 INFO:teuthology.nuke:targets: 2021-08-13T05:04:34.682 INFO:teuthology.kill: smithi114.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAwpAoCBZ6Acm24uWfAhjpInrJOjcsV+ZJKx1//BwKxLJh/o9GGodFPdVThg/7ZrgdoewEmqAxKULGwZm7pjBl3JqK4yfJLkBGs3vjR8RMQz8aFoDUCvV9PmHdHxrWA2VW1Wgso9O1YCR207qyr1n0Rzr6YSQBnQ3zKWv3WFGuwNTRYKq/abuqtZ0umhevI7zFhf6LZgD+722Mwsa7kXUeptTjmbbPQzR9pAVrw2TGZECheobOddcMFQ1Nm5yPSU3DS1KFG2ttbmTOzsQk3Tz160RMP47pZh6RD+vS46dRIjiXebZ7se2iujjZEQy2hLgB79wb4Bt1YWd7DK6iPGDixEeoZayyJcgvIStMRWW8hXvu52kDAjT391l869XPXNY2EioA/z76lbJrnWu0tz+FLjsfGOo/LnKSHqspZ4HPZqXyNJQFU+z7fb3XSY5M6LFPAhNkDHI7IbKZ4afdyCScU00mXsvBaQ/pURphmN3NxbCzvSw6zzf+m9nBZqqOn+c= 2021-08-13T05:04:34.682 INFO:teuthology.kill: smithi123.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDkBi52itkFZnySWCLMcvz30MJ2Hsv2BTYyM2qC8stFC2GRU4ygZ7iifYGxI/XY/hDJamKfZkgjJx8wiLERqH4ePEnFMi9TWnFpv8T8eW+nMhG8khBoQrpWKu9oTCJwmoBMJgECbL2BbsAMnedsOrYOhummWdPRaVo1YFG5BpuDB9r2Dv99+v2sCeooj5CCaYDryYYcG1RSG46I+qQgUqj/Yv8F7GawphjNpQ3x4ulkD2QTPmol6m1YBZR83rxdcS1w5whZD5ClizBh+Fz8NsBQKCcJfHLggUdmzw7u7w1XexlVrDmAeODlO22aTbqdjKjycsaDwOikpQS0vwDUwWc95W1jEDdndCsc7J/w9ThCZC65s0PYCrzDmfIpLxibe97Lz1w6ka6zEc5kBZe9d3pZz5wx5jAVZSepzYBtuKAedclMB8QDKui9vsxavluVy5Woo5go0OKLMSuAUP99v7eDM5JxhA3GsVpsE77jz6N8H0Wmd3vB1zrD3wkwil2eP58= 2021-08-13T05:04:34.682 INFO:teuthology.kill:2021-08-13 05:04:34,681.681 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-13T05:04:34.684 INFO:teuthology.kill:2021-08-13 05:04:34,684.684 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-13T05:04:34.743 INFO:teuthology.kill:2021-08-13 05:04:34,743.743 INFO:teuthology.orchestra.console:Power off smithi114 2021-08-13T05:04:34.760 INFO:teuthology.kill:2021-08-13 05:04:34,760.760 INFO:teuthology.orchestra.console:Power off smithi123 2021-08-13T05:04:43.122 INFO:teuthology.kill:2021-08-13 05:04:43,122.122 INFO:teuthology.orchestra.console:Power off for smithi123 completed 2021-08-13T05:04:47.229 INFO:teuthology.kill:2021-08-13 05:04:47,228.228 INFO:teuthology.orchestra.console:Power off for smithi114 completed 2021-08-13T05:04:47.451 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-08-13T05:04:47.451 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2021-08-13T05:04:47.452 INFO:teuthology.misc:Compressing logs... 2021-08-13T05:04:47.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-13T05:05:05.995 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-08-13T05:05:06.050 INFO:teuthology.kill:No teuthology processes running 2021-08-13T05:05:07.145 INFO:teuthology.kill:Nuking machines: ['smithi114', 'smithi123'] 2021-08-13T05:05:08.094 INFO:teuthology.kill:2021-08-13 05:05:08,093.093 INFO:teuthology.nuke:targets: 2021-08-13T05:05:08.094 INFO:teuthology.kill: smithi114.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAwpAoCBZ6Acm24uWfAhjpInrJOjcsV+ZJKx1//BwKxLJh/o9GGodFPdVThg/7ZrgdoewEmqAxKULGwZm7pjBl3JqK4yfJLkBGs3vjR8RMQz8aFoDUCvV9PmHdHxrWA2VW1Wgso9O1YCR207qyr1n0Rzr6YSQBnQ3zKWv3WFGuwNTRYKq/abuqtZ0umhevI7zFhf6LZgD+722Mwsa7kXUeptTjmbbPQzR9pAVrw2TGZECheobOddcMFQ1Nm5yPSU3DS1KFG2ttbmTOzsQk3Tz160RMP47pZh6RD+vS46dRIjiXebZ7se2iujjZEQy2hLgB79wb4Bt1YWd7DK6iPGDixEeoZayyJcgvIStMRWW8hXvu52kDAjT391l869XPXNY2EioA/z76lbJrnWu0tz+FLjsfGOo/LnKSHqspZ4HPZqXyNJQFU+z7fb3XSY5M6LFPAhNkDHI7IbKZ4afdyCScU00mXsvBaQ/pURphmN3NxbCzvSw6zzf+m9nBZqqOn+c= 2021-08-13T05:05:08.095 INFO:teuthology.kill: smithi123.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDkBi52itkFZnySWCLMcvz30MJ2Hsv2BTYyM2qC8stFC2GRU4ygZ7iifYGxI/XY/hDJamKfZkgjJx8wiLERqH4ePEnFMi9TWnFpv8T8eW+nMhG8khBoQrpWKu9oTCJwmoBMJgECbL2BbsAMnedsOrYOhummWdPRaVo1YFG5BpuDB9r2Dv99+v2sCeooj5CCaYDryYYcG1RSG46I+qQgUqj/Yv8F7GawphjNpQ3x4ulkD2QTPmol6m1YBZR83rxdcS1w5whZD5ClizBh+Fz8NsBQKCcJfHLggUdmzw7u7w1XexlVrDmAeODlO22aTbqdjKjycsaDwOikpQS0vwDUwWc95W1jEDdndCsc7J/w9ThCZC65s0PYCrzDmfIpLxibe97Lz1w6ka6zEc5kBZe9d3pZz5wx5jAVZSepzYBtuKAedclMB8QDKui9vsxavluVy5Woo5go0OKLMSuAUP99v7eDM5JxhA3GsVpsE77jz6N8H0Wmd3vB1zrD3wkwil2eP58= 2021-08-13T05:05:08.116 INFO:teuthology.kill:2021-08-13 05:05:08,116.116 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-13T05:05:08.119 INFO:teuthology.kill:2021-08-13 05:05:08,119.119 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-13T05:05:08.180 INFO:teuthology.kill:2021-08-13 05:05:08,180.180 INFO:teuthology.orchestra.console:Power off smithi114 2021-08-13T05:05:08.195 INFO:teuthology.kill:2021-08-13 05:05:08,195.195 INFO:teuthology.orchestra.console:Power off smithi123 2021-08-13T05:05:08.325 INFO:teuthology.kill:2021-08-13 05:05:08,325.325 INFO:teuthology.orchestra.console:Power off for smithi114 completed 2021-08-13T05:05:08.332 INFO:teuthology.kill:2021-08-13 05:05:08,332.332 INFO:teuthology.orchestra.console:Power off for smithi123 completed 2021-08-13T05:05:08.466 INFO:teuthology.kill:2021-08-13 05:05:08,466.466 INFO:teuthology.lock.ops:unlocked smithi114.front.sepia.ceph.com 2021-08-13T05:05:08.503 INFO:teuthology.kill:2021-08-13 05:05:08,503.503 INFO:teuthology.lock.ops:unlocked smithi123.front.sepia.ceph.com 2021-08-13T05:07:08.799 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-08-13T05:07:08.881 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-08-13T05:07:08.882 INFO:teuthology.nuke:Checking targets against current locks 2021-08-13T05:07:08.963 INFO:teuthology.nuke:Not nuking smithi114.front.sepia.ceph.com because description doesn't match 2021-08-13T05:07:08.963 INFO:teuthology.nuke:Not nuking smithi123.front.sepia.ceph.com because description doesn't match