2021-08-10T10:48:14.927 INFO:root:teuthology version: 1.1.0-321319b1 2021-08-10T10:48:15.090 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi037.front.sepia.ceph.com' reimaging 2021-08-10T10:48:15.092 INFO:teuthology.lock.ops:Updating [ubuntu@smithi037.front.sepia.ceph.com]: reset os type and version on server 2021-08-10T10:48:15.092 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2021-08-10T10:48:15.110 INFO:teuthology.lock.ops:Node 'ubuntu@smithi037.front.sepia.ceph.com' reimaging is complete 2021-08-10T10:48:15.111 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi051.front.sepia.ceph.com' reimaging 2021-08-10T10:48:15.111 INFO:teuthology.lock.ops:Updating [ubuntu@smithi051.front.sepia.ceph.com]: reset os type and version on server 2021-08-10T10:48:15.111 INFO:teuthology.lock.ops:Updating smithi051.front.sepia.ceph.com on lock server 2021-08-10T10:48:15.132 INFO:teuthology.lock.ops:Node 'ubuntu@smithi051.front.sepia.ceph.com' reimaging is complete 2021-08-10T10:48:15.323 INFO:teuthology.provision.fog.smithi051:Scheduling deploy of ubuntu 20.04 2021-08-10T10:48:15.324 INFO:teuthology.provision.fog.smithi037:Scheduling deploy of ubuntu 20.04 2021-08-10T10:48:15.630 INFO:teuthology.orchestra.console:Power off smithi037 2021-08-10T10:48:15.644 INFO:teuthology.orchestra.console:Power off smithi051 2021-08-10T10:48:15.773 INFO:teuthology.orchestra.console:Power off for smithi037 completed 2021-08-10T10:48:15.786 INFO:teuthology.orchestra.console:Power off for smithi051 completed 2021-08-10T10:48:15.875 INFO:teuthology.orchestra.console:Power on smithi037 2021-08-10T10:48:15.887 INFO:teuthology.orchestra.console:Power on smithi051 2021-08-10T10:48:24.245 INFO:teuthology.orchestra.console:Power on for smithi037 completed 2021-08-10T10:48:24.250 INFO:teuthology.orchestra.console:Power on for smithi051 completed 2021-08-10T10:48:24.347 INFO:teuthology.provision.fog.smithi037:Waiting for deploy to finish 2021-08-10T10:48:24.352 INFO:teuthology.provision.fog.smithi051:Waiting for deploy to finish 2021-08-10T10:52:52.943 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-10T10:52:53.452 INFO:teuthology.provision.fog.smithi037:Node is ready 2021-08-10T10:52:53.459 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037.front.sepia.ceph.com 2021-08-10T10:52:53.466 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37 smithi037.front.sepia.ceph.com smithi037 2021-08-10T10:52:53.649 INFO:teuthology.provision.fog.smithi037:Deploy complete! 2021-08-10T10:53:01.954 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-10T10:53:02.445 INFO:teuthology.provision.fog.smithi051:Node is ready 2021-08-10T10:53:02.451 INFO:teuthology.orchestra.run.smithi051.stdout:smithi051.front.sepia.ceph.com 2021-08-10T10:53:02.459 INFO:teuthology.orchestra.run.smithi051.stdout:172.21.15.51 smithi051.front.sepia.ceph.com smithi051 2021-08-10T10:53:02.612 INFO:teuthology.provision.fog.smithi051:Deploy complete! 2021-08-10T10:53:02.786 INFO:teuthology.lock.ops:Checking smithi037.front.sepia.ceph.com 2021-08-10T10:53:02.787 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-10T10:53:02.803 INFO:teuthology.lock.ops:Checking smithi051.front.sepia.ceph.com 2021-08-10T10:53:02.804 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-10T10:53:02.820 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: set os type and version on server 2021-08-10T10:53:02.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-10T10:53:02.998 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2021-08-10T10:53:03.005 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="Ubuntu" 2021-08-10T10:53:03.005 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-08-10T10:53:03.005 INFO:teuthology.orchestra.run.smithi037.stdout:ID=ubuntu 2021-08-10T10:53:03.006 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE=debian 2021-08-10T10:53:03.006 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-08-10T10:53:03.006 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="20.04" 2021-08-10T10:53:03.006 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://www.ubuntu.com/" 2021-08-10T10:53:03.007 INFO:teuthology.orchestra.run.smithi037.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-08-10T10:53:03.007 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-08-10T10:53:03.007 INFO:teuthology.orchestra.run.smithi037.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-08-10T10:53:03.007 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_CODENAME=focal 2021-08-10T10:53:03.007 INFO:teuthology.orchestra.run.smithi037.stdout:UBUNTU_CODENAME=focal 2021-08-10T10:53:03.008 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2021-08-10T10:53:03.024 INFO:teuthology.lock.ops:Updating [smithi051.front.sepia.ceph.com]: set os type and version on server 2021-08-10T10:53:03.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-10T10:53:03.196 INFO:teuthology.orchestra.run.smithi051.stdout:x86_64 2021-08-10T10:53:03.202 INFO:teuthology.orchestra.run.smithi051.stdout:NAME="Ubuntu" 2021-08-10T10:53:03.202 INFO:teuthology.orchestra.run.smithi051.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-08-10T10:53:03.203 INFO:teuthology.orchestra.run.smithi051.stdout:ID=ubuntu 2021-08-10T10:53:03.203 INFO:teuthology.orchestra.run.smithi051.stdout:ID_LIKE=debian 2021-08-10T10:53:03.203 INFO:teuthology.orchestra.run.smithi051.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-08-10T10:53:03.203 INFO:teuthology.orchestra.run.smithi051.stdout:VERSION_ID="20.04" 2021-08-10T10:53:03.204 INFO:teuthology.orchestra.run.smithi051.stdout:HOME_URL="https://www.ubuntu.com/" 2021-08-10T10:53:03.204 INFO:teuthology.orchestra.run.smithi051.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-08-10T10:53:03.204 INFO:teuthology.orchestra.run.smithi051.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-08-10T10:53:03.204 INFO:teuthology.orchestra.run.smithi051.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-08-10T10:53:03.204 INFO:teuthology.orchestra.run.smithi051.stdout:VERSION_CODENAME=focal 2021-08-10T10:53:03.205 INFO:teuthology.orchestra.run.smithi051.stdout:UBUNTU_CODENAME=focal 2021-08-10T10:53:03.205 INFO:teuthology.lock.ops:Updating smithi051.front.sepia.ceph.com on lock server 2021-08-10T10:53:03.281 INFO:teuthology.dispatcher.supervisor:Running job 6329690 2021-08-10T10:53:03.281 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-08-09_20:42:46-rados-wip-yuri3-testing-2021-08-09-1006-distro-basic-smithi/6329690 --name yuriw-2021-08-09_20:42:46-rados-wip-yuri3-testing-2021-08-09-1006-distro-basic-smithi --description rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-start-upgrade 4-wait mon_election/connectivity} -- /home/teuthworker/archive/yuriw-2021-08-09_20:42:46-rados-wip-yuri3-testing-2021-08-09-1006-distro-basic-smithi/6329690/orig.config.yaml 2021-08-10T10:53:03.285 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-08-09_20:42:46-rados-wip-yuri3-testing-2021-08-09-1006-distro-basic-smithi/6329690 2021-08-10T10:53:03.286 INFO:teuthology.dispatcher.supervisor:Job PID: 22013 2021-08-10T10:53:03.286 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-08-10T22:53:53.999 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-08-10T22:53:54.208 INFO:teuthology.kill:Killing Pids: {22013} 2021-08-10T22:53:55.393 INFO:teuthology.kill:Nuking machines: ['smithi037', 'smithi051'] 2021-08-10T22:53:56.408 INFO:teuthology.kill:2021-08-10 22:53:56,407.407 INFO:teuthology.nuke:targets: 2021-08-10T22:53:56.409 INFO:teuthology.kill: smithi037.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtgPsPqxP8FHbpLCYT8E6kucPHS1dUwT+uVRGFIIW/ML6FaPU4Q01b7KaSb/NkGRXFWTi6XW2zLK+CcBes5LrzT+0VpDbZ+jJ3/PtN5CFyvMDUUz6JPU+PnIlLCyBjzN84XozcuZqX4SjzoWphHJJ8ngNxci9FHDr+vc880y+KO8AzG3lFxCVTkr9GKCzJlqTNNqnito2xNnv7Svuz7iYsVX1Y9bkeu/67VdiUqqwi2zJPXTdJDM5ysdEPUPCbPrxmlGsXIErFO/4k96Spjs8Ip5tTQRg93X+SApde5+x11iXKHFvv5mYKV+k/1kqx2L7Q1bSOTu8OXFLVrJLa7H1e30L52WpP2DMxTP0eEO4RxotJXQItIWnFvwFtj+1ZAD2LLEYF0ukSdVInruz7J7Tw17py7OETKaBSeQHPXUpTf/crI9RrmIhq/Mg39SL2919L/z3zDLeydGzja/wRspTA6dpG2C9Z7CB6fWzh7boN35NfaiippCRi1c+imjCm08M= 2021-08-10T22:53:56.409 INFO:teuthology.kill: smithi051.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDO9Fa+Bx+h3bTkJDn3TL/NiyX7rSGI5tObxVx9ypUgzw5krLNAabLiWum6yynVWjKYiGv78alta6+wgGT2mGdSTxbFdh4kdu9iVKb83OrefJKe679w1AxUPqo50FIAMLxC1VbI2gc+6FbnCo+pbXlYDKRQ1o87f+E+DkH0pq659Fz09uP0eq8tGQgWJoMeJqvm4bYT3586b5hF5sCEayOPC+tH3tXd6AzeBCQ6HDrgFBdqz4Wbb2wGhqpt8vH5f0ssBJ4qNnrO+HOfMCAxeCqpD+u/3XRhmJ4os7zvCI4Nz8mAWKiM6DXPI60565S6DMwjNZQ61eKybqoxHPIpx6X5Zfg6pwNAZBkdnxm25pvNJTS7CpvE9wjFMyKno3eEQDycefFAlskGn33O4Hww5N2kYuAIAP3iUUPk6hC+x4//3dspKb7M9C2lEde8gS/2wCt0T9KjIUepflVPMZZr22cJvVCN/fdNgD2y4b6c3hZcLtzwQgmtOcLTE5xYx7eROeM= 2021-08-10T22:53:56.409 INFO:teuthology.kill:2021-08-10 22:53:56,408.408 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-10T22:53:56.410 INFO:teuthology.kill:2021-08-10 22:53:56,410.410 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-10T22:53:56.470 INFO:teuthology.kill:2021-08-10 22:53:56,469.469 INFO:teuthology.orchestra.console:Power off smithi051 2021-08-10T22:53:56.509 INFO:teuthology.kill:2021-08-10 22:53:56,509.509 INFO:teuthology.orchestra.console:Power off smithi037 2021-08-10T22:54:04.844 INFO:teuthology.kill:2021-08-10 22:54:04,844.844 INFO:teuthology.orchestra.console:Power off for smithi051 completed 2021-08-10T22:54:04.873 INFO:teuthology.kill:2021-08-10 22:54:04,872.872 INFO:teuthology.orchestra.console:Power off for smithi037 completed 2021-08-10T22:54:05.112 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2021-08-10T22:54:05.113 INFO:teuthology.task.internal:roles: ubuntu@smithi051.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2021-08-10T22:54:05.113 INFO:teuthology.misc:Compressing logs... 2021-08-10T22:54:05.114 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-10T22:55:05.175 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-08-10T22:55:05.219 INFO:teuthology.kill:No teuthology processes running 2021-08-10T22:55:06.336 INFO:teuthology.kill:Nuking machines: ['smithi037', 'smithi051'] 2021-08-10T22:55:07.358 INFO:teuthology.kill:2021-08-10 22:55:07,358.358 INFO:teuthology.nuke:targets: 2021-08-10T22:55:07.359 INFO:teuthology.kill: smithi037.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtgPsPqxP8FHbpLCYT8E6kucPHS1dUwT+uVRGFIIW/ML6FaPU4Q01b7KaSb/NkGRXFWTi6XW2zLK+CcBes5LrzT+0VpDbZ+jJ3/PtN5CFyvMDUUz6JPU+PnIlLCyBjzN84XozcuZqX4SjzoWphHJJ8ngNxci9FHDr+vc880y+KO8AzG3lFxCVTkr9GKCzJlqTNNqnito2xNnv7Svuz7iYsVX1Y9bkeu/67VdiUqqwi2zJPXTdJDM5ysdEPUPCbPrxmlGsXIErFO/4k96Spjs8Ip5tTQRg93X+SApde5+x11iXKHFvv5mYKV+k/1kqx2L7Q1bSOTu8OXFLVrJLa7H1e30L52WpP2DMxTP0eEO4RxotJXQItIWnFvwFtj+1ZAD2LLEYF0ukSdVInruz7J7Tw17py7OETKaBSeQHPXUpTf/crI9RrmIhq/Mg39SL2919L/z3zDLeydGzja/wRspTA6dpG2C9Z7CB6fWzh7boN35NfaiippCRi1c+imjCm08M= 2021-08-10T22:55:07.359 INFO:teuthology.kill: smithi051.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDO9Fa+Bx+h3bTkJDn3TL/NiyX7rSGI5tObxVx9ypUgzw5krLNAabLiWum6yynVWjKYiGv78alta6+wgGT2mGdSTxbFdh4kdu9iVKb83OrefJKe679w1AxUPqo50FIAMLxC1VbI2gc+6FbnCo+pbXlYDKRQ1o87f+E+DkH0pq659Fz09uP0eq8tGQgWJoMeJqvm4bYT3586b5hF5sCEayOPC+tH3tXd6AzeBCQ6HDrgFBdqz4Wbb2wGhqpt8vH5f0ssBJ4qNnrO+HOfMCAxeCqpD+u/3XRhmJ4os7zvCI4Nz8mAWKiM6DXPI60565S6DMwjNZQ61eKybqoxHPIpx6X5Zfg6pwNAZBkdnxm25pvNJTS7CpvE9wjFMyKno3eEQDycefFAlskGn33O4Hww5N2kYuAIAP3iUUPk6hC+x4//3dspKb7M9C2lEde8gS/2wCt0T9KjIUepflVPMZZr22cJvVCN/fdNgD2y4b6c3hZcLtzwQgmtOcLTE5xYx7eROeM= 2021-08-10T22:55:07.381 INFO:teuthology.kill:2021-08-10 22:55:07,380.380 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-10T22:55:07.384 INFO:teuthology.kill:2021-08-10 22:55:07,384.384 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-10T22:55:07.444 INFO:teuthology.kill:2021-08-10 22:55:07,443.443 INFO:teuthology.orchestra.console:Power off smithi051 2021-08-10T22:55:07.459 INFO:teuthology.kill:2021-08-10 22:55:07,459.459 INFO:teuthology.orchestra.console:Power off smithi037 2021-08-10T22:55:07.590 INFO:teuthology.kill:2021-08-10 22:55:07,590.590 INFO:teuthology.orchestra.console:Power off for smithi051 completed 2021-08-10T22:55:07.604 INFO:teuthology.kill:2021-08-10 22:55:07,604.604 INFO:teuthology.orchestra.console:Power off for smithi037 completed 2021-08-10T22:55:07.730 INFO:teuthology.kill:2021-08-10 22:55:07,729.729 INFO:teuthology.lock.ops:unlocked smithi051.front.sepia.ceph.com 2021-08-10T22:55:07.745 INFO:teuthology.kill:2021-08-10 22:55:07,745.745 INFO:teuthology.lock.ops:unlocked smithi037.front.sepia.ceph.com 2021-08-10T22:57:08.050 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-08-10T22:57:08.144 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-08-10T22:57:08.145 INFO:teuthology.nuke:Checking targets against current locks 2021-08-10T22:57:08.248 INFO:teuthology.nuke:Not nuking smithi051.front.sepia.ceph.com because description doesn't match 2021-08-10T22:57:08.249 INFO:teuthology.nuke:Not nuking smithi037.front.sepia.ceph.com because description doesn't match