2021-03-12T11:30:32.927 INFO:root:teuthology version: 1.1.0-e8438aef 2021-03-12T11:30:33.021 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi057.front.sepia.ceph.com' reimaging 2021-03-12T11:30:33.022 INFO:teuthology.lock.ops:Updating [ubuntu@smithi057.front.sepia.ceph.com]: reset os type and version on server 2021-03-12T11:30:33.022 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2021-03-12T11:30:33.037 INFO:teuthology.lock.ops:Node 'ubuntu@smithi057.front.sepia.ceph.com' reimaging is complete 2021-03-12T11:30:33.038 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi176.front.sepia.ceph.com' reimaging 2021-03-12T11:30:33.038 INFO:teuthology.lock.ops:Updating [ubuntu@smithi176.front.sepia.ceph.com]: reset os type and version on server 2021-03-12T11:30:33.038 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2021-03-12T11:30:33.057 INFO:teuthology.lock.ops:Node 'ubuntu@smithi176.front.sepia.ceph.com' reimaging is complete 2021-03-12T11:30:33.240 INFO:teuthology.provision.fog.smithi057:Scheduling deploy of ubuntu 20.04 2021-03-12T11:30:33.249 INFO:teuthology.provision.fog.smithi176:Scheduling deploy of ubuntu 20.04 2021-03-12T11:30:33.548 INFO:teuthology.orchestra.console:Power off smithi057 2021-03-12T11:30:33.591 INFO:teuthology.orchestra.console:Power off smithi176 2021-03-12T11:30:41.954 INFO:teuthology.orchestra.console:Power off for smithi176 completed 2021-03-12T11:30:42.055 INFO:teuthology.orchestra.console:Power on smithi176 2021-03-12T11:30:46.039 INFO:teuthology.orchestra.console:Power off for smithi057 completed 2021-03-12T11:30:46.141 INFO:teuthology.orchestra.console:Power on smithi057 2021-03-12T11:30:50.399 INFO:teuthology.orchestra.console:Power on for smithi057 completed 2021-03-12T11:30:50.423 INFO:teuthology.orchestra.console:Power on for smithi176 completed 2021-03-12T11:30:50.501 INFO:teuthology.provision.fog.smithi057:Waiting for deploy to finish 2021-03-12T11:30:50.524 INFO:teuthology.provision.fog.smithi176:Waiting for deploy to finish 2021-03-12T11:34:36.842 ERROR:teuthology.orchestra.connection:Error authenticating with smithi176.front.sepia.ceph.com: Authentication timeout. 2021-03-12T11:35:24.354 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2021-03-12T11:35:39.758 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-12T11:35:40.251 INFO:teuthology.provision.fog.smithi176:Node is ready 2021-03-12T11:35:40.263 INFO:teuthology.orchestra.run.smithi176.stdout:smithi176.front.sepia.ceph.com 2021-03-12T11:35:40.318 INFO:teuthology.orchestra.run.smithi176.stdout:172.21.15.176 smithi176.front.sepia.ceph.com smithi176 2021-03-12T11:35:40.530 INFO:teuthology.provision.fog.smithi176:Deploy complete! 2021-03-12T11:35:46.348 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-12T11:35:46.943 INFO:teuthology.provision.fog.smithi057:Node is ready 2021-03-12T11:35:46.950 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057.front.sepia.ceph.com 2021-03-12T11:35:46.957 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57 smithi057.front.sepia.ceph.com smithi057 2021-03-12T11:35:47.089 INFO:teuthology.provision.fog.smithi057:Deploy complete! 2021-03-12T11:35:47.281 INFO:teuthology.lock.ops:Checking smithi057.front.sepia.ceph.com 2021-03-12T11:35:47.288 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-12T11:35:47.305 INFO:teuthology.lock.ops:Checking smithi176.front.sepia.ceph.com 2021-03-12T11:35:47.305 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-12T11:35:47.322 INFO:teuthology.lock.ops:Updating [smithi057.front.sepia.ceph.com]: set os type and version on server 2021-03-12T11:35:47.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-12T11:35:47.498 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64 2021-03-12T11:35:47.504 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="Ubuntu" 2021-03-12T11:35:47.505 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-03-12T11:35:47.505 INFO:teuthology.orchestra.run.smithi057.stdout:ID=ubuntu 2021-03-12T11:35:47.505 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE=debian 2021-03-12T11:35:47.505 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-03-12T11:35:47.506 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="20.04" 2021-03-12T11:35:47.506 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-12T11:35:47.506 INFO:teuthology.orchestra.run.smithi057.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-12T11:35:47.506 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-12T11:35:47.507 INFO:teuthology.orchestra.run.smithi057.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-12T11:35:47.507 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_CODENAME=focal 2021-03-12T11:35:47.507 INFO:teuthology.orchestra.run.smithi057.stdout:UBUNTU_CODENAME=focal 2021-03-12T11:35:47.508 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2021-03-12T11:35:47.524 INFO:teuthology.lock.ops:Updating [smithi176.front.sepia.ceph.com]: set os type and version on server 2021-03-12T11:35:47.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-12T11:35:47.706 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2021-03-12T11:35:47.712 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="Ubuntu" 2021-03-12T11:35:47.713 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-03-12T11:35:47.713 INFO:teuthology.orchestra.run.smithi176.stdout:ID=ubuntu 2021-03-12T11:35:47.713 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE=debian 2021-03-12T11:35:47.713 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-03-12T11:35:47.714 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="20.04" 2021-03-12T11:35:47.714 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-12T11:35:47.714 INFO:teuthology.orchestra.run.smithi176.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-12T11:35:47.714 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-12T11:35:47.715 INFO:teuthology.orchestra.run.smithi176.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-12T11:35:47.715 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_CODENAME=focal 2021-03-12T11:35:47.715 INFO:teuthology.orchestra.run.smithi176.stdout:UBUNTU_CODENAME=focal 2021-03-12T11:35:47.715 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2021-03-12T11:35:47.965 INFO:teuthology.dispatcher.supervisor:Running job 5958530 2021-03-12T11:35:47.966 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/bin/teuthology -v --owner scheduled_swagner@teuthology --archive /home/teuthworker/archive/swagner-2021-03-12_11:13:04-rados:cephadm:upgrade-wip-swagner-testing-2021-03-11-1320-distro-basic-smithi/5958530 --name swagner-2021-03-12_11:13:04-rados:cephadm:upgrade-wip-swagner-testing-2021-03-11-1320-distro-basic-smithi --description rados:cephadm:upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-start-upgrade 4-wait fixed-2 mon_election/connectivity} -- /home/teuthworker/archive/swagner-2021-03-12_11:13:04-rados:cephadm:upgrade-wip-swagner-testing-2021-03-11-1320-distro-basic-smithi/5958530/orig.config.yaml 2021-03-12T11:35:47.970 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/swagner-2021-03-12_11:13:04-rados:cephadm:upgrade-wip-swagner-testing-2021-03-11-1320-distro-basic-smithi/5958530 2021-03-12T11:35:47.971 INFO:teuthology.dispatcher.supervisor:Job PID: 4227 2021-03-12T11:35:47.972 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-03-12T23:36:05.247 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-12T23:36:05.773 INFO:teuthology.kill:Killing Pids: {4227} 2021-03-12T23:36:07.107 INFO:teuthology.kill:Nuking machines: ['smithi057', 'smithi176'] 2021-03-12T23:36:08.148 INFO:teuthology.kill:2021-03-12 23:36:08,148.148 INFO:teuthology.nuke:targets: 2021-03-12T23:36:08.149 INFO:teuthology.kill: smithi057.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7oeNbOKQAgkOm0tV/Ot7vEu3UsW2X8Ir85JDIYRq+OZ/IKtMXgWTKHOOmQpm9A/DDrc0zZTWTtqz8rIesjJIij31emaAMPtrhP5ktlPr2Ql0WfGK7mZciQyu66hXpiP54nHQ5LnB8i41sRkzwjtGS0dqGvjqiMf8Bru8Fcmjdfq3MEKd1EU95Cbns2OjliDRBtghIjk5FF2S0dndF4YbJEaLFZFmIrSMZ7isehzQSyNW6lqSmYiPpt/MUkHtrONMU6fjB1KwQfA6UH9ssg+XKBqeO7hq/k10ftxdax6wG5KGHxjMlfJBhNgtbSKX67EZzMxCZsqDHqZ/PjK1F8wXz+S5ivzTJ8qzELOhArMFsM2NS1U9/YPwiSmhieU+3aVFuv8rAy9I6OgxYQANsW3Xy+kOdvMd/6/jYhhKz6rT9mp7IgkeVJ+08k43POxBAkRKFxyjpy5vLS7fyfCGAxQ4PZeA1oHXl2XvRhkHyXz2szaujwuRL7609J7KdLJ5SoBU= 2021-03-12T23:36:08.151 INFO:teuthology.kill: smithi176.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4jzLtN5H56LwS7PrzvlUCaRW1VShgYGrJM/uUDPVpg346pxvhgzc3PLHh2oex2n1CxdMG/EGeJrmlnJj69I8cAh4VveDgr1X3I4hw+SEuGwijyPiPRA5VpIFlWQVUzPTd/HPxk9iaPXssFpMtWcelsPgzm3XmnRdtCpPuCXaKOuwAELkQYkYBOb4KnJeZ7z4Eoz/O/td4JF1zvw/kAeF2WNjdr9V1oxdw2PFggMx6xGIhrw0QfGgme8EW7IRm4RfBzkYIYvRpYTZK5AgaERGpM/FxWXtNaJQokUP4rzIi3rKW2MNcKvjnPKK4pziCk6x0XFbCBy2ijRsNWuRPmO8Y2j+nzFfoh+D2A4OzJj1kCQBXTtQWlOZPP1/HkKGV/2ohkIUNC7nckYcPglgcrRP4udZbHU4q84xV2P/KhEPpy7hdHHEj5QdmnWeUopK8596Whw3DVV2GeWFQJ9SUj5EZLZ2lX4h824gQJgja9w70z/Z35OVmnsmzd7NtDP/+h08= 2021-03-12T23:36:08.151 INFO:teuthology.kill:2021-03-12 23:36:08,148.148 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-12T23:36:08.152 INFO:teuthology.kill:2021-03-12 23:36:08,151.151 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-12T23:36:08.215 INFO:teuthology.kill:2021-03-12 23:36:08,215.215 INFO:teuthology.orchestra.console:Power off smithi057 2021-03-12T23:36:08.235 INFO:teuthology.kill:2021-03-12 23:36:08,235.235 INFO:teuthology.orchestra.console:Power off smithi176 2021-03-12T23:36:16.599 INFO:teuthology.kill:2021-03-12 23:36:16,599.599 INFO:teuthology.orchestra.console:Power off for smithi176 completed 2021-03-12T23:36:20.708 INFO:teuthology.kill:2021-03-12 23:36:20,707.707 INFO:teuthology.orchestra.console:Power off for smithi057 completed 2021-03-12T23:36:20.998 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2021-03-12T23:36:21.003 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2021-03-12T23:36:21.010 INFO:teuthology.misc:Compressing logs... 2021-03-12T23:36:21.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-12T23:36:55.757 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-12T23:36:55.808 INFO:teuthology.kill:No teuthology processes running 2021-03-12T23:36:56.864 INFO:teuthology.kill:Nuking machines: ['smithi057', 'smithi176'] 2021-03-12T23:36:57.785 INFO:teuthology.kill:2021-03-12 23:36:57,784.784 INFO:teuthology.nuke:targets: 2021-03-12T23:36:57.786 INFO:teuthology.kill: smithi057.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7oeNbOKQAgkOm0tV/Ot7vEu3UsW2X8Ir85JDIYRq+OZ/IKtMXgWTKHOOmQpm9A/DDrc0zZTWTtqz8rIesjJIij31emaAMPtrhP5ktlPr2Ql0WfGK7mZciQyu66hXpiP54nHQ5LnB8i41sRkzwjtGS0dqGvjqiMf8Bru8Fcmjdfq3MEKd1EU95Cbns2OjliDRBtghIjk5FF2S0dndF4YbJEaLFZFmIrSMZ7isehzQSyNW6lqSmYiPpt/MUkHtrONMU6fjB1KwQfA6UH9ssg+XKBqeO7hq/k10ftxdax6wG5KGHxjMlfJBhNgtbSKX67EZzMxCZsqDHqZ/PjK1F8wXz+S5ivzTJ8qzELOhArMFsM2NS1U9/YPwiSmhieU+3aVFuv8rAy9I6OgxYQANsW3Xy+kOdvMd/6/jYhhKz6rT9mp7IgkeVJ+08k43POxBAkRKFxyjpy5vLS7fyfCGAxQ4PZeA1oHXl2XvRhkHyXz2szaujwuRL7609J7KdLJ5SoBU= 2021-03-12T23:36:57.786 INFO:teuthology.kill: smithi176.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4jzLtN5H56LwS7PrzvlUCaRW1VShgYGrJM/uUDPVpg346pxvhgzc3PLHh2oex2n1CxdMG/EGeJrmlnJj69I8cAh4VveDgr1X3I4hw+SEuGwijyPiPRA5VpIFlWQVUzPTd/HPxk9iaPXssFpMtWcelsPgzm3XmnRdtCpPuCXaKOuwAELkQYkYBOb4KnJeZ7z4Eoz/O/td4JF1zvw/kAeF2WNjdr9V1oxdw2PFggMx6xGIhrw0QfGgme8EW7IRm4RfBzkYIYvRpYTZK5AgaERGpM/FxWXtNaJQokUP4rzIi3rKW2MNcKvjnPKK4pziCk6x0XFbCBy2ijRsNWuRPmO8Y2j+nzFfoh+D2A4OzJj1kCQBXTtQWlOZPP1/HkKGV/2ohkIUNC7nckYcPglgcrRP4udZbHU4q84xV2P/KhEPpy7hdHHEj5QdmnWeUopK8596Whw3DVV2GeWFQJ9SUj5EZLZ2lX4h824gQJgja9w70z/Z35OVmnsmzd7NtDP/+h08= 2021-03-12T23:36:57.806 INFO:teuthology.kill:2021-03-12 23:36:57,806.806 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-12T23:36:57.809 INFO:teuthology.kill:2021-03-12 23:36:57,809.809 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-12T23:36:57.865 INFO:teuthology.kill:2021-03-12 23:36:57,865.865 INFO:teuthology.orchestra.console:Power off smithi176 2021-03-12T23:36:57.880 INFO:teuthology.kill:2021-03-12 23:36:57,879.879 INFO:teuthology.orchestra.console:Power off smithi057 2021-03-12T23:36:58.020 INFO:teuthology.kill:2021-03-12 23:36:58,020.020 INFO:teuthology.orchestra.console:Power off for smithi176 completed 2021-03-12T23:36:58.026 INFO:teuthology.kill:2021-03-12 23:36:58,025.025 INFO:teuthology.orchestra.console:Power off for smithi057 completed 2021-03-12T23:36:58.168 INFO:teuthology.kill:2021-03-12 23:36:58,167.167 INFO:teuthology.lock.ops:unlocked smithi176.front.sepia.ceph.com 2021-03-12T23:36:58.171 INFO:teuthology.kill:2021-03-12 23:36:58,171.171 INFO:teuthology.lock.ops:unlocked smithi057.front.sepia.ceph.com 2021-03-12T23:38:58.462 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-03-12T23:38:58.755 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-03-12T23:38:58.757 INFO:teuthology.nuke:Checking targets against current locks 2021-03-12T23:38:58.922 INFO:teuthology.nuke:Not nuking smithi057.front.sepia.ceph.com because description doesn't match 2021-03-12T23:38:58.923 INFO:teuthology.nuke:Not nuking smithi176.front.sepia.ceph.com because description doesn't match