2021-03-12T11:30:13.546 INFO:root:teuthology version: 1.1.0-e8438aef 2021-03-12T11:30:13.689 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi039.front.sepia.ceph.com' reimaging 2021-03-12T11:30:13.691 INFO:teuthology.lock.ops:Updating [ubuntu@smithi039.front.sepia.ceph.com]: reset os type and version on server 2021-03-12T11:30:13.691 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2021-03-12T11:30:13.710 INFO:teuthology.lock.ops:Node 'ubuntu@smithi039.front.sepia.ceph.com' reimaging is complete 2021-03-12T11:30:13.711 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi091.front.sepia.ceph.com' reimaging 2021-03-12T11:30:13.711 INFO:teuthology.lock.ops:Updating [ubuntu@smithi091.front.sepia.ceph.com]: reset os type and version on server 2021-03-12T11:30:13.712 INFO:teuthology.lock.ops:Updating smithi091.front.sepia.ceph.com on lock server 2021-03-12T11:30:13.732 INFO:teuthology.lock.ops:Node 'ubuntu@smithi091.front.sepia.ceph.com' reimaging is complete 2021-03-12T11:30:13.930 INFO:teuthology.provision.fog.smithi039:Scheduling deploy of ubuntu 20.04 2021-03-12T11:30:13.933 INFO:teuthology.provision.fog.smithi091:Scheduling deploy of ubuntu 20.04 2021-03-12T11:30:14.258 INFO:teuthology.orchestra.console:Power off smithi091 2021-03-12T11:30:14.287 INFO:teuthology.orchestra.console:Power off smithi039 2021-03-12T11:30:22.696 INFO:teuthology.orchestra.console:Power off for smithi039 completed 2021-03-12T11:30:22.798 INFO:teuthology.orchestra.console:Power on smithi039 2021-03-12T11:30:26.799 INFO:teuthology.orchestra.console:Power off for smithi091 completed 2021-03-12T11:30:26.902 INFO:teuthology.orchestra.console:Power on smithi091 2021-03-12T11:30:31.164 INFO:teuthology.orchestra.console:Power on for smithi091 completed 2021-03-12T11:30:31.199 INFO:teuthology.orchestra.console:Power on for smithi039 completed 2021-03-12T11:30:31.266 INFO:teuthology.provision.fog.smithi091:Waiting for deploy to finish 2021-03-12T11:30:31.301 INFO:teuthology.provision.fog.smithi039:Waiting for deploy to finish 2021-03-12T11:35:29.669 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-12T11:35:30.147 INFO:teuthology.provision.fog.smithi091:Node is ready 2021-03-12T11:35:30.195 INFO:teuthology.orchestra.run.smithi091.stdout:smithi091.front.sepia.ceph.com 2021-03-12T11:35:30.203 INFO:teuthology.orchestra.run.smithi091.stdout:172.21.15.91 smithi091.front.sepia.ceph.com smithi091 2021-03-12T11:35:30.340 INFO:teuthology.provision.fog.smithi091:Deploy complete! 2021-03-12T11:35:52.082 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-12T11:35:52.544 INFO:teuthology.provision.fog.smithi039:Node is ready 2021-03-12T11:35:52.551 INFO:teuthology.orchestra.run.smithi039.stdout:smithi039.front.sepia.ceph.com 2021-03-12T11:35:52.556 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39 smithi039.front.sepia.ceph.com smithi039 2021-03-12T11:35:52.801 INFO:teuthology.provision.fog.smithi039:Deploy complete! 2021-03-12T11:35:52.965 INFO:teuthology.lock.ops:Checking smithi039.front.sepia.ceph.com 2021-03-12T11:35:52.969 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-12T11:35:52.984 INFO:teuthology.lock.ops:Checking smithi091.front.sepia.ceph.com 2021-03-12T11:35:52.984 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-12T11:35:52.998 INFO:teuthology.lock.ops:Updating [smithi039.front.sepia.ceph.com]: set os type and version on server 2021-03-12T11:35:52.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-12T11:35:53.173 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2021-03-12T11:35:53.180 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="Ubuntu" 2021-03-12T11:35:53.180 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-03-12T11:35:53.180 INFO:teuthology.orchestra.run.smithi039.stdout:ID=ubuntu 2021-03-12T11:35:53.180 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE=debian 2021-03-12T11:35:53.181 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-03-12T11:35:53.181 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="20.04" 2021-03-12T11:35:53.181 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-12T11:35:53.181 INFO:teuthology.orchestra.run.smithi039.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-12T11:35:53.182 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-12T11:35:53.182 INFO:teuthology.orchestra.run.smithi039.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-12T11:35:53.182 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_CODENAME=focal 2021-03-12T11:35:53.182 INFO:teuthology.orchestra.run.smithi039.stdout:UBUNTU_CODENAME=focal 2021-03-12T11:35:53.183 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2021-03-12T11:35:53.200 INFO:teuthology.lock.ops:Updating [smithi091.front.sepia.ceph.com]: set os type and version on server 2021-03-12T11:35:53.201 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-12T11:35:53.466 INFO:teuthology.orchestra.run.smithi091.stdout:x86_64 2021-03-12T11:35:53.514 INFO:teuthology.orchestra.run.smithi091.stdout:NAME="Ubuntu" 2021-03-12T11:35:53.515 INFO:teuthology.orchestra.run.smithi091.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-03-12T11:35:53.515 INFO:teuthology.orchestra.run.smithi091.stdout:ID=ubuntu 2021-03-12T11:35:53.515 INFO:teuthology.orchestra.run.smithi091.stdout:ID_LIKE=debian 2021-03-12T11:35:53.516 INFO:teuthology.orchestra.run.smithi091.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-03-12T11:35:53.516 INFO:teuthology.orchestra.run.smithi091.stdout:VERSION_ID="20.04" 2021-03-12T11:35:53.516 INFO:teuthology.orchestra.run.smithi091.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-12T11:35:53.516 INFO:teuthology.orchestra.run.smithi091.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-12T11:35:53.516 INFO:teuthology.orchestra.run.smithi091.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-12T11:35:53.517 INFO:teuthology.orchestra.run.smithi091.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-12T11:35:53.517 INFO:teuthology.orchestra.run.smithi091.stdout:VERSION_CODENAME=focal 2021-03-12T11:35:53.517 INFO:teuthology.orchestra.run.smithi091.stdout:UBUNTU_CODENAME=focal 2021-03-12T11:35:53.518 INFO:teuthology.lock.ops:Updating smithi091.front.sepia.ceph.com on lock server 2021-03-12T11:35:53.593 INFO:teuthology.dispatcher.supervisor:Running job 5958529 2021-03-12T11:35:53.593 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/bin/teuthology -v --owner scheduled_swagner@teuthology --archive /home/teuthworker/archive/swagner-2021-03-12_11:13:04-rados:cephadm:upgrade-wip-swagner-testing-2021-03-11-1320-distro-basic-smithi/5958529 --name swagner-2021-03-12_11:13:04-rados:cephadm:upgrade-wip-swagner-testing-2021-03-11-1320-distro-basic-smithi --description rados:cephadm:upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-start-upgrade 4-wait fixed-2 mon_election/classic} -- /home/teuthworker/archive/swagner-2021-03-12_11:13:04-rados:cephadm:upgrade-wip-swagner-testing-2021-03-11-1320-distro-basic-smithi/5958529/orig.config.yaml 2021-03-12T11:35:53.597 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/swagner-2021-03-12_11:13:04-rados:cephadm:upgrade-wip-swagner-testing-2021-03-11-1320-distro-basic-smithi/5958529 2021-03-12T11:35:53.598 INFO:teuthology.dispatcher.supervisor:Job PID: 4533 2021-03-12T11:35:53.598 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-03-12T23:36:10.276 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-12T23:36:10.831 INFO:teuthology.kill:Killing Pids: {4533} 2021-03-12T23:36:12.223 INFO:teuthology.kill:Nuking machines: ['smithi039', 'smithi091'] 2021-03-12T23:36:13.156 INFO:teuthology.kill:2021-03-12 23:36:13,155.155 INFO:teuthology.nuke:targets: 2021-03-12T23:36:13.158 INFO:teuthology.kill: smithi039.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgMOORZ4yuO8Z2T6nDpgQuE1TGKgMkw34bpI583WkqEgh+HQf+M7jZhNRf6wijqKLwF+LnkZ80ldcQ69IWSwYbgLNLotCo8E9szuRwCRTXp1aNsi6LTb6RxvsTXbEbfQqVAohqRvVvdGUXooPg2q0nfKBZJkaC732m3nQwUq2oCc159H8eTVZIRXk3tcRLgVnaP942WqODtkgXw3VWLbTkpZt+f69Bq31DPHp3+HEizaf78lP9I/H7JpUvZIU9l9ydPn9tASySfe5u8avYpRVtYJ0F2iydV5mfPz4K+WgxyiG2/KrBvJyh6Jk3bDVYuZ6cPof3WGgU0wMXcnJNKCEWU51XZuxoKfUdwqaQMPDoVDfSjYFyMO2dmIoF3CT4oIFvT2IgA8nRCT+HhrBy+825Vmc/Hoi90aAtPNL11UVz8T7t7yALuPFfGJ9/GCSP5FSbxGj1n04ABylTRa0MqWo/asInj2J18F9ZGTGGZo017wKe6kesgFz75Tjc0rf+yQk= 2021-03-12T23:36:13.159 INFO:teuthology.kill: smithi091.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDX69pGbqpLBqcGTCZRYnWAenpBYxtw7T/H5hQK1SfkjQnBwyhPvBsPNXuTpPG3LxsDrLoEmUwwfGxno01UzBwPc4tpacBjYPsu3KCVrH9pd6BXtotkly+x/NOxeBJs5CqFnQzH27fyMPxpJbkcWCl7kC6rKRFEXbHBEcs4NvZyUOz0JF7ySdE6oG2Y6MLoJrTjcuwAl9uDQniLWURQu4eFoPszggPsCASpa6P2oVUPfd8BPsWWeSgCVCHZ7k5fzSocVUYA16GlUGzc6OI3tIs6l/OlrFQVGUtkyy/F4benGNnABV6j9JkTr8bvF8NxZH0WfUdGSfu+A/6cpmIYfX6yV22mhY+TQjcMtRfKBCPLT2EmeAOxMHXYd1ZS6zAN/SxCGLZ/O3IHxu9rl0SSeVjsX38aBJ0qHr7EwgHZy883zpNtSp1YsqVVleACQSATo4XfFNliB5RIQ8yLoqJXmscbyKxbcGJsU7xgFfwpBS/ZjG7EvsBgxtcE9AlmXFJUlDE= 2021-03-12T23:36:13.160 INFO:teuthology.kill:2021-03-12 23:36:13,155.155 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-12T23:36:13.160 INFO:teuthology.kill:2021-03-12 23:36:13,157.157 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-12T23:36:13.211 INFO:teuthology.kill:2021-03-12 23:36:13,211.211 INFO:teuthology.orchestra.console:Power off smithi039 2021-03-12T23:36:13.225 INFO:teuthology.kill:2021-03-12 23:36:13,225.225 INFO:teuthology.orchestra.console:Power off smithi091 2021-03-12T23:36:21.581 INFO:teuthology.kill:2021-03-12 23:36:21,581.581 INFO:teuthology.orchestra.console:Power off for smithi039 completed 2021-03-12T23:36:25.717 INFO:teuthology.kill:2021-03-12 23:36:25,716.716 INFO:teuthology.orchestra.console:Power off for smithi091 completed 2021-03-12T23:36:25.996 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2021-03-12T23:36:26.002 INFO:teuthology.task.internal:roles: ubuntu@smithi091.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2021-03-12T23:36:26.008 INFO:teuthology.misc:Compressing logs... 2021-03-12T23:36:26.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-12T23:37:26.191 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-12T23:37:26.290 INFO:teuthology.kill:No teuthology processes running 2021-03-12T23:37:27.348 INFO:teuthology.kill:Nuking machines: ['smithi039', 'smithi091'] 2021-03-12T23:37:28.265 INFO:teuthology.kill:2021-03-12 23:37:28,264.264 INFO:teuthology.nuke:targets: 2021-03-12T23:37:28.265 INFO:teuthology.kill: smithi039.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgMOORZ4yuO8Z2T6nDpgQuE1TGKgMkw34bpI583WkqEgh+HQf+M7jZhNRf6wijqKLwF+LnkZ80ldcQ69IWSwYbgLNLotCo8E9szuRwCRTXp1aNsi6LTb6RxvsTXbEbfQqVAohqRvVvdGUXooPg2q0nfKBZJkaC732m3nQwUq2oCc159H8eTVZIRXk3tcRLgVnaP942WqODtkgXw3VWLbTkpZt+f69Bq31DPHp3+HEizaf78lP9I/H7JpUvZIU9l9ydPn9tASySfe5u8avYpRVtYJ0F2iydV5mfPz4K+WgxyiG2/KrBvJyh6Jk3bDVYuZ6cPof3WGgU0wMXcnJNKCEWU51XZuxoKfUdwqaQMPDoVDfSjYFyMO2dmIoF3CT4oIFvT2IgA8nRCT+HhrBy+825Vmc/Hoi90aAtPNL11UVz8T7t7yALuPFfGJ9/GCSP5FSbxGj1n04ABylTRa0MqWo/asInj2J18F9ZGTGGZo017wKe6kesgFz75Tjc0rf+yQk= 2021-03-12T23:37:28.268 INFO:teuthology.kill: smithi091.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDX69pGbqpLBqcGTCZRYnWAenpBYxtw7T/H5hQK1SfkjQnBwyhPvBsPNXuTpPG3LxsDrLoEmUwwfGxno01UzBwPc4tpacBjYPsu3KCVrH9pd6BXtotkly+x/NOxeBJs5CqFnQzH27fyMPxpJbkcWCl7kC6rKRFEXbHBEcs4NvZyUOz0JF7ySdE6oG2Y6MLoJrTjcuwAl9uDQniLWURQu4eFoPszggPsCASpa6P2oVUPfd8BPsWWeSgCVCHZ7k5fzSocVUYA16GlUGzc6OI3tIs6l/OlrFQVGUtkyy/F4benGNnABV6j9JkTr8bvF8NxZH0WfUdGSfu+A/6cpmIYfX6yV22mhY+TQjcMtRfKBCPLT2EmeAOxMHXYd1ZS6zAN/SxCGLZ/O3IHxu9rl0SSeVjsX38aBJ0qHr7EwgHZy883zpNtSp1YsqVVleACQSATo4XfFNliB5RIQ8yLoqJXmscbyKxbcGJsU7xgFfwpBS/ZjG7EvsBgxtcE9AlmXFJUlDE= 2021-03-12T23:37:28.286 INFO:teuthology.kill:2021-03-12 23:37:28,286.286 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-12T23:37:28.289 INFO:teuthology.kill:2021-03-12 23:37:28,289.289 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-12T23:37:28.347 INFO:teuthology.kill:2021-03-12 23:37:28,347.347 INFO:teuthology.orchestra.console:Power off smithi039 2021-03-12T23:37:28.359 INFO:teuthology.kill:2021-03-12 23:37:28,359.359 INFO:teuthology.orchestra.console:Power off smithi091 2021-03-12T23:37:28.487 INFO:teuthology.kill:2021-03-12 23:37:28,486.486 INFO:teuthology.orchestra.console:Power off for smithi039 completed 2021-03-12T23:37:28.496 INFO:teuthology.kill:2021-03-12 23:37:28,496.496 INFO:teuthology.orchestra.console:Power off for smithi091 completed 2021-03-12T23:37:28.628 INFO:teuthology.kill:2021-03-12 23:37:28,627.627 INFO:teuthology.lock.ops:unlocked smithi039.front.sepia.ceph.com 2021-03-12T23:37:28.634 INFO:teuthology.kill:2021-03-12 23:37:28,634.634 INFO:teuthology.lock.ops:unlocked smithi091.front.sepia.ceph.com 2021-03-12T23:39:28.902 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-03-12T23:39:29.009 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-03-12T23:39:29.014 INFO:teuthology.nuke:Checking targets against current locks 2021-03-12T23:39:29.252 INFO:teuthology.nuke:Not nuking smithi039.front.sepia.ceph.com because description doesn't match 2021-03-12T23:39:29.254 INFO:teuthology.nuke:Not nuking smithi091.front.sepia.ceph.com because description doesn't match