2021-08-11T15:32:18.365 INFO:root:teuthology version: 1.1.0-321319b1 2021-08-11T15:32:18.463 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi060.front.sepia.ceph.com' reimaging 2021-08-11T15:32:18.464 INFO:teuthology.lock.ops:Updating [ubuntu@smithi060.front.sepia.ceph.com]: reset os type and version on server 2021-08-11T15:32:18.464 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2021-08-11T15:32:18.482 INFO:teuthology.lock.ops:Node 'ubuntu@smithi060.front.sepia.ceph.com' reimaging is complete 2021-08-11T15:32:18.483 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi125.front.sepia.ceph.com' reimaging 2021-08-11T15:32:18.483 INFO:teuthology.lock.ops:Updating [ubuntu@smithi125.front.sepia.ceph.com]: reset os type and version on server 2021-08-11T15:32:18.484 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server 2021-08-11T15:32:18.502 INFO:teuthology.lock.ops:Node 'ubuntu@smithi125.front.sepia.ceph.com' reimaging is complete 2021-08-11T15:32:18.693 INFO:teuthology.provision.fog.smithi060:Scheduling deploy of ubuntu 20.04 2021-08-11T15:32:18.741 INFO:teuthology.provision.fog.smithi125:Scheduling deploy of ubuntu 20.04 2021-08-11T15:32:19.027 INFO:teuthology.orchestra.console:Power off smithi060 2021-08-11T15:32:19.072 INFO:teuthology.orchestra.console:Power off smithi125 2021-08-11T15:32:19.181 INFO:teuthology.orchestra.console:Power off for smithi060 completed 2021-08-11T15:32:19.213 INFO:teuthology.orchestra.console:Power off for smithi125 completed 2021-08-11T15:32:19.282 INFO:teuthology.orchestra.console:Power on smithi060 2021-08-11T15:32:19.315 INFO:teuthology.orchestra.console:Power on smithi125 2021-08-11T15:32:23.537 INFO:teuthology.orchestra.console:Power on for smithi060 completed 2021-08-11T15:32:23.639 INFO:teuthology.provision.fog.smithi060:Waiting for deploy to finish 2021-08-11T15:32:27.686 INFO:teuthology.orchestra.console:Power on for smithi125 completed 2021-08-11T15:32:27.788 INFO:teuthology.provision.fog.smithi125:Waiting for deploy to finish 2021-08-11T15:37:01.357 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-11T15:37:01.840 INFO:teuthology.provision.fog.smithi060:Node is ready 2021-08-11T15:37:01.847 INFO:teuthology.orchestra.run.smithi060.stdout:smithi060.front.sepia.ceph.com 2021-08-11T15:37:01.896 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60 smithi060.front.sepia.ceph.com smithi060 2021-08-11T15:37:02.062 INFO:teuthology.provision.fog.smithi060:Deploy complete! 2021-08-11T15:37:21.987 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-11T15:37:22.447 INFO:teuthology.provision.fog.smithi125:Node is ready 2021-08-11T15:37:22.495 INFO:teuthology.orchestra.run.smithi125.stdout:smithi125.front.sepia.ceph.com 2021-08-11T15:37:22.502 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125 smithi125.front.sepia.ceph.com smithi125 2021-08-11T15:37:22.646 INFO:teuthology.provision.fog.smithi125:Deploy complete! 2021-08-11T15:37:22.810 INFO:teuthology.lock.ops:Checking smithi060.front.sepia.ceph.com 2021-08-11T15:37:22.811 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-11T15:37:22.827 INFO:teuthology.lock.ops:Checking smithi125.front.sepia.ceph.com 2021-08-11T15:37:22.828 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-11T15:37:22.841 INFO:teuthology.lock.ops:Updating [smithi060.front.sepia.ceph.com]: set os type and version on server 2021-08-11T15:37:22.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-11T15:37:23.138 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2021-08-11T15:37:23.146 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="Ubuntu" 2021-08-11T15:37:23.146 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-08-11T15:37:23.146 INFO:teuthology.orchestra.run.smithi060.stdout:ID=ubuntu 2021-08-11T15:37:23.147 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE=debian 2021-08-11T15:37:23.147 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-08-11T15:37:23.147 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="20.04" 2021-08-11T15:37:23.147 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://www.ubuntu.com/" 2021-08-11T15:37:23.148 INFO:teuthology.orchestra.run.smithi060.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-08-11T15:37:23.148 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-08-11T15:37:23.148 INFO:teuthology.orchestra.run.smithi060.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-08-11T15:37:23.148 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_CODENAME=focal 2021-08-11T15:37:23.149 INFO:teuthology.orchestra.run.smithi060.stdout:UBUNTU_CODENAME=focal 2021-08-11T15:37:23.150 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2021-08-11T15:37:23.165 INFO:teuthology.lock.ops:Updating [smithi125.front.sepia.ceph.com]: set os type and version on server 2021-08-11T15:37:23.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-11T15:37:23.371 INFO:teuthology.orchestra.run.smithi125.stdout:x86_64 2021-08-11T15:37:23.378 INFO:teuthology.orchestra.run.smithi125.stdout:NAME="Ubuntu" 2021-08-11T15:37:23.378 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-08-11T15:37:23.379 INFO:teuthology.orchestra.run.smithi125.stdout:ID=ubuntu 2021-08-11T15:37:23.379 INFO:teuthology.orchestra.run.smithi125.stdout:ID_LIKE=debian 2021-08-11T15:37:23.379 INFO:teuthology.orchestra.run.smithi125.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-08-11T15:37:23.379 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION_ID="20.04" 2021-08-11T15:37:23.379 INFO:teuthology.orchestra.run.smithi125.stdout:HOME_URL="https://www.ubuntu.com/" 2021-08-11T15:37:23.380 INFO:teuthology.orchestra.run.smithi125.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-08-11T15:37:23.380 INFO:teuthology.orchestra.run.smithi125.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-08-11T15:37:23.380 INFO:teuthology.orchestra.run.smithi125.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-08-11T15:37:23.380 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION_CODENAME=focal 2021-08-11T15:37:23.381 INFO:teuthology.orchestra.run.smithi125.stdout:UBUNTU_CODENAME=focal 2021-08-11T15:37:23.381 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server 2021-08-11T15:37:23.455 INFO:teuthology.dispatcher.supervisor:Running job 6334768 2021-08-11T15:37:23.455 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-08-11_14:15:52-rados-wip-yuri3-testing-2021-08-09-1006-distro-basic-smithi/6334768 --name yuriw-2021-08-11_14:15:52-rados-wip-yuri3-testing-2021-08-09-1006-distro-basic-smithi --description rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-start-upgrade 4-wait mon_election/classic} -- /home/teuthworker/archive/yuriw-2021-08-11_14:15:52-rados-wip-yuri3-testing-2021-08-09-1006-distro-basic-smithi/6334768/orig.config.yaml 2021-08-11T15:37:23.459 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-08-11_14:15:52-rados-wip-yuri3-testing-2021-08-09-1006-distro-basic-smithi/6334768 2021-08-11T15:37:23.460 INFO:teuthology.dispatcher.supervisor:Job PID: 18449 2021-08-11T15:37:23.460 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-08-12T03:38:13.423 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-08-12T03:38:13.668 INFO:teuthology.kill:Killing Pids: {18449} 2021-08-12T03:38:14.785 INFO:teuthology.kill:Nuking machines: ['smithi060', 'smithi125'] 2021-08-12T03:38:15.822 INFO:teuthology.kill:2021-08-12 03:38:15,822.822 INFO:teuthology.nuke:targets: 2021-08-12T03:38:15.823 INFO:teuthology.kill: smithi060.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKLpVkLVdDjqorqSSrkf6zLGK7ZK9V3cOKcHvXLd+IkZktVxk7tfVmLAftlDu/SKMi/+/oTvUXgLsXWjWOWbSjuHZ5wfpGVaqMIalW2g7R3amvSvYP1eDxk3ftWezANgZqHSHhoD5296Q9cdke+Yh9c0HVfxYSE3qOfKT0rwlpJCfU/gpMq/h08U35YUxVZ7u4XxD6dJuDyStP6qwt/VkPJCCG69lW3bTqlznWEjqpOmWf6uAJhMYfKlVuSjg6cc9lTF+e0CTc3cigrk0e/u4D6WatCV78WVCuWeigiLaC0LEjgor0ggyC5W5bkFDvdw01051xdHCCp45bmuzHooA+j60aGvUJmBUUp8QlQY7+iGrLPpx8dY+UVYNMMYHL+99yJDTFYKQDMzB67PgUHKi7WDnpn9ln9aHVSchGxe7Cg5/TobHlBJimMmP+2ji0e2i2Uk5PPpo+aI223WYeIBW6BjtYJCGW0vm02f9oOQ9SgQkI8AlyXY5zdKnXr2SDN10= 2021-08-12T03:38:15.823 INFO:teuthology.kill: smithi125.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCp3SHSsvmIcN3kQ+w8RRU0CAmRuBdUnjvA8ENkpOTBLrQr+ReRWthHCwDuSzFGqc5GH5TUoYmuO6ZY3CWfnoA4uKrgcctIKP2vR8uwQXeO0FWqBxyymWSnGJz2j2fSS0HMJS3I/uZ1NOG2rhLhE9nKJMTKX3RcEQlVJh/002dl+3syamoewDddL42LEsnVpUlDr1Zq6RUMKhJeGYpWt+l3INEB5X8y4MStTUnEbWq57ZSd/c5BMNUMJCLkLCO43p4hK4/B1beJmFlH1uCtR4+rk4gtGjVzIfEUFwhMK3SRKM0FN40luoTkQTUvmT97GM499hwt6cwbF4coHd8aGxX92Gy0rLRUblJUqVF+6HbEr0ddsyYDwkDWQ9EgYj2/83PIPdcmYXBbTYljaVGv6eeyCCma3oePJGx49SWyliZymsroRX/JlfDxBtRV/vS8FmmkIYZO45WrS1HN5k5JCL+AnT4RQr7WN9gPBaJleOt8U9noaCMtz7yiDNHBULV28Jc= 2021-08-12T03:38:15.824 INFO:teuthology.kill:2021-08-12 03:38:15,822.822 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-12T03:38:15.826 INFO:teuthology.kill:2021-08-12 03:38:15,826.826 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-12T03:38:15.885 INFO:teuthology.kill:2021-08-12 03:38:15,885.885 INFO:teuthology.orchestra.console:Power off smithi125 2021-08-12T03:38:15.900 INFO:teuthology.kill:2021-08-12 03:38:15,900.900 INFO:teuthology.orchestra.console:Power off smithi060 2021-08-12T03:38:24.262 INFO:teuthology.kill:2021-08-12 03:38:24,261.261 INFO:teuthology.orchestra.console:Power off for smithi125 completed 2021-08-12T03:38:28.376 INFO:teuthology.kill:2021-08-12 03:38:28,376.376 INFO:teuthology.orchestra.console:Power off for smithi060 completed 2021-08-12T03:38:28.597 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2021-08-12T03:38:28.597 INFO:teuthology.task.internal:roles: ubuntu@smithi125.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2021-08-12T03:38:28.598 INFO:teuthology.misc:Compressing logs... 2021-08-12T03:38:28.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-12T03:39:28.663 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-08-12T03:39:28.706 INFO:teuthology.kill:No teuthology processes running 2021-08-12T03:39:29.792 INFO:teuthology.kill:Nuking machines: ['smithi060', 'smithi125'] 2021-08-12T03:39:30.757 INFO:teuthology.kill:2021-08-12 03:39:30,757.757 INFO:teuthology.nuke:targets: 2021-08-12T03:39:30.758 INFO:teuthology.kill: smithi060.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKLpVkLVdDjqorqSSrkf6zLGK7ZK9V3cOKcHvXLd+IkZktVxk7tfVmLAftlDu/SKMi/+/oTvUXgLsXWjWOWbSjuHZ5wfpGVaqMIalW2g7R3amvSvYP1eDxk3ftWezANgZqHSHhoD5296Q9cdke+Yh9c0HVfxYSE3qOfKT0rwlpJCfU/gpMq/h08U35YUxVZ7u4XxD6dJuDyStP6qwt/VkPJCCG69lW3bTqlznWEjqpOmWf6uAJhMYfKlVuSjg6cc9lTF+e0CTc3cigrk0e/u4D6WatCV78WVCuWeigiLaC0LEjgor0ggyC5W5bkFDvdw01051xdHCCp45bmuzHooA+j60aGvUJmBUUp8QlQY7+iGrLPpx8dY+UVYNMMYHL+99yJDTFYKQDMzB67PgUHKi7WDnpn9ln9aHVSchGxe7Cg5/TobHlBJimMmP+2ji0e2i2Uk5PPpo+aI223WYeIBW6BjtYJCGW0vm02f9oOQ9SgQkI8AlyXY5zdKnXr2SDN10= 2021-08-12T03:39:30.758 INFO:teuthology.kill: smithi125.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCp3SHSsvmIcN3kQ+w8RRU0CAmRuBdUnjvA8ENkpOTBLrQr+ReRWthHCwDuSzFGqc5GH5TUoYmuO6ZY3CWfnoA4uKrgcctIKP2vR8uwQXeO0FWqBxyymWSnGJz2j2fSS0HMJS3I/uZ1NOG2rhLhE9nKJMTKX3RcEQlVJh/002dl+3syamoewDddL42LEsnVpUlDr1Zq6RUMKhJeGYpWt+l3INEB5X8y4MStTUnEbWq57ZSd/c5BMNUMJCLkLCO43p4hK4/B1beJmFlH1uCtR4+rk4gtGjVzIfEUFwhMK3SRKM0FN40luoTkQTUvmT97GM499hwt6cwbF4coHd8aGxX92Gy0rLRUblJUqVF+6HbEr0ddsyYDwkDWQ9EgYj2/83PIPdcmYXBbTYljaVGv6eeyCCma3oePJGx49SWyliZymsroRX/JlfDxBtRV/vS8FmmkIYZO45WrS1HN5k5JCL+AnT4RQr7WN9gPBaJleOt8U9noaCMtz7yiDNHBULV28Jc= 2021-08-12T03:39:30.777 INFO:teuthology.kill:2021-08-12 03:39:30,777.777 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-12T03:39:30.780 INFO:teuthology.kill:2021-08-12 03:39:30,780.780 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-12T03:39:30.835 INFO:teuthology.kill:2021-08-12 03:39:30,835.835 INFO:teuthology.orchestra.console:Power off smithi060 2021-08-12T03:39:30.851 INFO:teuthology.kill:2021-08-12 03:39:30,850.850 INFO:teuthology.orchestra.console:Power off smithi125 2021-08-12T03:39:30.980 INFO:teuthology.kill:2021-08-12 03:39:30,980.980 INFO:teuthology.orchestra.console:Power off for smithi060 completed 2021-08-12T03:39:30.990 INFO:teuthology.kill:2021-08-12 03:39:30,990.990 INFO:teuthology.orchestra.console:Power off for smithi125 completed 2021-08-12T03:39:31.121 INFO:teuthology.kill:2021-08-12 03:39:31,120.120 INFO:teuthology.lock.ops:unlocked smithi060.front.sepia.ceph.com 2021-08-12T03:39:31.144 INFO:teuthology.kill:2021-08-12 03:39:31,143.143 INFO:teuthology.lock.ops:unlocked smithi125.front.sepia.ceph.com 2021-08-12T03:41:31.431 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-08-12T03:41:31.513 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-08-12T03:41:31.513 INFO:teuthology.nuke:Checking targets against current locks 2021-08-12T03:41:31.599 INFO:teuthology.nuke:Not nuking smithi060.front.sepia.ceph.com because description doesn't match 2021-08-12T03:41:31.600 INFO:teuthology.nuke:Not nuking smithi125.front.sepia.ceph.com because description doesn't match