2021-05-18T22:48:25.791 INFO:root:teuthology version: 1.1.0-e1e1a4bf
2021-05-18T22:48:25.884 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi105.front.sepia.ceph.com' reimaging
2021-05-18T22:48:25.885 INFO:teuthology.lock.ops:Updating [ubuntu@smithi105.front.sepia.ceph.com]: reset os type and version on server
2021-05-18T22:48:25.886 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server
2021-05-18T22:48:25.899 INFO:teuthology.lock.ops:Node 'ubuntu@smithi105.front.sepia.ceph.com' reimaging is complete
2021-05-18T22:48:25.899 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi114.front.sepia.ceph.com' reimaging
2021-05-18T22:48:25.899 INFO:teuthology.lock.ops:Updating [ubuntu@smithi114.front.sepia.ceph.com]: reset os type and version on server
2021-05-18T22:48:25.900 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server
2021-05-18T22:48:25.913 INFO:teuthology.lock.ops:Node 'ubuntu@smithi114.front.sepia.ceph.com' reimaging is complete
2021-05-18T22:48:26.097 INFO:teuthology.provision.fog.smithi105:Scheduling deploy of ubuntu 20.04
2021-05-18T22:48:26.110 INFO:teuthology.provision.fog.smithi114:Scheduling deploy of ubuntu 20.04
2021-05-18T22:48:26.437 INFO:teuthology.orchestra.console:Power off smithi114
2021-05-18T22:48:26.450 INFO:teuthology.orchestra.console:Power off smithi105
2021-05-18T22:48:34.818 INFO:teuthology.orchestra.console:Power off for smithi105 completed
2021-05-18T22:48:34.919 INFO:teuthology.orchestra.console:Power on smithi105
2021-05-18T22:48:38.921 INFO:teuthology.orchestra.console:Power off for smithi114 completed
2021-05-18T22:48:39.023 INFO:teuthology.orchestra.console:Power on smithi114
2021-05-18T22:48:43.277 INFO:teuthology.orchestra.console:Power on for smithi114 completed
2021-05-18T22:48:43.282 INFO:teuthology.orchestra.console:Power on for smithi105 completed
2021-05-18T22:48:43.378 INFO:teuthology.provision.fog.smithi114:Waiting for deploy to finish
2021-05-18T22:48:43.383 INFO:teuthology.provision.fog.smithi105:Waiting for deploy to finish
2021-05-18T22:53:37.752 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-05-18T22:53:38.200 INFO:teuthology.provision.fog.smithi114:Node is ready
2021-05-18T22:53:38.207 INFO:teuthology.orchestra.run.smithi114.stdout:smithi114.front.sepia.ceph.com
2021-05-18T22:53:38.215 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114 smithi114.front.sepia.ceph.com smithi114
2021-05-18T22:53:38.432 INFO:teuthology.provision.fog.smithi114:Deploy complete!
2021-05-18T22:53:43.730 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-05-18T22:53:44.146 INFO:teuthology.provision.fog.smithi105:Node is ready
2021-05-18T22:53:44.152 INFO:teuthology.orchestra.run.smithi105.stdout:smithi105.front.sepia.ceph.com
2021-05-18T22:53:44.160 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.15.105 smithi105.front.sepia.ceph.com smithi105
2021-05-18T22:53:44.395 INFO:teuthology.provision.fog.smithi105:Deploy complete!
2021-05-18T22:53:44.553 INFO:teuthology.lock.ops:Checking smithi105.front.sepia.ceph.com
2021-05-18T22:53:44.554 INFO:teuthology.lock.ops:New key found. Updating...
2021-05-18T22:53:44.664 INFO:teuthology.lock.ops:Checking smithi114.front.sepia.ceph.com
2021-05-18T22:53:44.665 INFO:teuthology.lock.ops:New key found. Updating...
2021-05-18T22:53:44.733 INFO:teuthology.lock.ops:Updating [smithi105.front.sepia.ceph.com]: set os type and version on server
2021-05-18T22:53:44.733 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-05-18T22:53:44.907 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64
2021-05-18T22:53:44.912 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="Ubuntu"
2021-05-18T22:53:44.912 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="20.04.2 LTS (Focal Fossa)"
2021-05-18T22:53:44.913 INFO:teuthology.orchestra.run.smithi105.stdout:ID=ubuntu
2021-05-18T22:53:44.913 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE=debian
2021-05-18T22:53:44.913 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS"
2021-05-18T22:53:44.913 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="20.04"
2021-05-18T22:53:44.914 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://www.ubuntu.com/"
2021-05-18T22:53:44.914 INFO:teuthology.orchestra.run.smithi105.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-05-18T22:53:44.914 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-05-18T22:53:44.914 INFO:teuthology.orchestra.run.smithi105.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-05-18T22:53:44.915 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_CODENAME=focal
2021-05-18T22:53:44.915 INFO:teuthology.orchestra.run.smithi105.stdout:UBUNTU_CODENAME=focal
2021-05-18T22:53:44.916 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server
2021-05-18T22:53:44.946 INFO:teuthology.lock.ops:Updating [smithi114.front.sepia.ceph.com]: set os type and version on server
2021-05-18T22:53:44.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-05-18T22:53:45.114 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64
2021-05-18T22:53:45.120 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="Ubuntu"
2021-05-18T22:53:45.120 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="20.04.2 LTS (Focal Fossa)"
2021-05-18T22:53:45.120 INFO:teuthology.orchestra.run.smithi114.stdout:ID=ubuntu
2021-05-18T22:53:45.121 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE=debian
2021-05-18T22:53:45.121 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS"
2021-05-18T22:53:45.121 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="20.04"
2021-05-18T22:53:45.121 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://www.ubuntu.com/"
2021-05-18T22:53:45.122 INFO:teuthology.orchestra.run.smithi114.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-05-18T22:53:45.122 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-05-18T22:53:45.122 INFO:teuthology.orchestra.run.smithi114.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-05-18T22:53:45.122 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_CODENAME=focal
2021-05-18T22:53:45.122 INFO:teuthology.orchestra.run.smithi114.stdout:UBUNTU_CODENAME=focal
2021-05-18T22:53:45.123 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server
2021-05-18T22:53:45.310 INFO:teuthology.dispatcher.supervisor:Running job 6120504
2021-05-18T22:53:45.311 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_e1e1a4bfc0b70ac61ab556b3b9a694948c7a0089/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2021-05-18_16:18:58-rados-wip_gbenhano_ncb2-distro-basic-smithi/6120504 --name benhanokh-2021-05-18_16:18:58-rados-wip_gbenhano_ncb2-distro-basic-smithi --description rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-start-upgrade 4-wait mon_election/connectivity} -- /home/teuthworker/archive/benhanokh-2021-05-18_16:18:58-rados-wip_gbenhano_ncb2-distro-basic-smithi/6120504/orig.config.yaml
2021-05-18T22:53:45.314 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2021-05-18_16:18:58-rados-wip_gbenhano_ncb2-distro-basic-smithi/6120504
2021-05-18T22:53:45.315 INFO:teuthology.dispatcher.supervisor:Job PID: 15526
2021-05-18T22:53:45.315 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-05-19T10:00:30.833 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/benhanokh-2021-05-18_16:18:58-rados-wip_gbenhano_ncb2-distro-basic-smithi/jobs/ failed with status 500:
Internal Server Error
Internal Server Error
2021-05-19T10:54:35.568 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-05-19T10:54:35.797 INFO:teuthology.kill:Killing Pids: {15526}
2021-05-19T10:54:36.870 INFO:teuthology.kill:Nuking machines: ['smithi105', 'smithi114']
2021-05-19T10:54:37.784 INFO:teuthology.kill:2021-05-19 10:54:37,782.782 INFO:teuthology.nuke:targets:
2021-05-19T10:54:37.785 INFO:teuthology.kill: smithi105.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8bbfbLIUHGzwdk+eFknNLqJ2a0d6TEtaqJegPU6S7zPAGSA3ONEhR9UDIGz4RiS2ci7dhyXjZzPQAEQxnvsW6vPLlR/hg3ssEaPuxYcj5gWyjQ6ziVNeLtDKjcC5laAajd+J7Ic0J3HZEjmtDQf9PUNv/w7FAK/7kP5DDcU5cOUqrxpmK/AB7ONmh2GehBgP4tYB8A6x81+Zik56HvQ2EsaOdjDrG51g/YdRPtnmt/yte8tQG0nEHZ4FUWTAwXVCHycm1PRVyY1rpeoCSBQY3cFnv90WpZtPtwfC8QC+bJkfM6y7m5spGQt966dGb1/8o1l2I/ICAru92zFOIWnD4VGmvT2K4ZvRXNbKZo4rn4U83qgrS6Foy1QxRP8oMkOzXURvzNPs/7f/y4pq27BX/+a+5cva6GfdkH4fu9oUZlQxuPkat/rEle3O0+CoyEDT/HVg4f6zSQI5tZfyYgdxVHBo7OrO7EfwRc6WmtcXFt5HFC4sJoayqMViucSZdt6c=
2021-05-19T10:54:37.785 INFO:teuthology.kill: smithi114.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCcf20IWWVnhlPtoP/CHmRfsJrBWUFQU2f/KtHE6GofZuqj3GxN/mN5yFtQmvpEfNWK59Ze2vY8nV+ObeUPw+zAw7YFIimasq2oBebaVIi7bGouGb7QksajEgy1OUUivddzYrxrJOFebzBZgqAE5lwokKMq/PtAxPh4ccj++DXKZOYo/l97aWvybYX8XcWXy3XF5xUXjHRLNJ4bGuu0xtxJyKcs6wcfw6QlMFDIdJ1cnwYwjLZki0nj7wZdBvXhDgTiuu/FZjzOK1YxqmcrM+paFHZfgFtnB8iFdxJiPkAGeSpJB2Ll9tCsxYqxdf2OUgm+ww7nt420AzbFy8q5odePDFppRr2wSKE8LVQ20y1+tgjgJLvjjiu5yDZnBVDTLl3TNbHgMOihfeShEIdZumvEPUZUU67KEmX+L+MBkmAgkOE4yq0jb+FylU71mSkyPHGmutLtaaX+558dcH2xfW0vZvxGp08asgP/MXd1by0BhEDokhm4L1LpHgzledF2JTU=
2021-05-19T10:54:37.786 INFO:teuthology.kill:2021-05-19 10:54:37,783.783 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-19T10:54:37.786 INFO:teuthology.kill:2021-05-19 10:54:37,785.785 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-19T10:54:37.851 INFO:teuthology.kill:2021-05-19 10:54:37,850.850 INFO:teuthology.orchestra.console:Power off smithi114
2021-05-19T10:54:37.866 INFO:teuthology.kill:2021-05-19 10:54:37,865.865 INFO:teuthology.orchestra.console:Power off smithi105
2021-05-19T10:54:46.225 INFO:teuthology.kill:2021-05-19 10:54:46,225.225 INFO:teuthology.orchestra.console:Power off for smithi114 completed
2021-05-19T10:54:46.232 INFO:teuthology.kill:2021-05-19 10:54:46,232.232 INFO:teuthology.orchestra.console:Power off for smithi105 completed
2021-05-19T10:54:46.470 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']
2021-05-19T10:54:46.471 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a']
2021-05-19T10:54:46.471 INFO:teuthology.misc:Compressing logs...
2021-05-19T10:54:46.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-05-19T10:55:46.535 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_e1e1a4bfc0b70ac61ab556b3b9a694948c7a0089/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
teuth_config.archive_base, job_config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_e1e1a4bfc0b70ac61ab556b3b9a694948c7a0089/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
compress_logs(ctx, log_path)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_e1e1a4bfc0b70ac61ab556b3b9a694948c7a0089/teuthology/misc.py", line 1378, in compress_logs
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_e1e1a4bfc0b70ac61ab556b3b9a694948c7a0089/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_e1e1a4bfc0b70ac61ab556b3b9a694948c7a0089/teuthology/orchestra/cluster.py", line 64, in
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_e1e1a4bfc0b70ac61ab556b3b9a694948c7a0089/teuthology/orchestra/remote.py", line 509, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_e1e1a4bfc0b70ac61ab556b3b9a694948c7a0089/teuthology/orchestra/run.py", line 434, in run
transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-05-19T10:55:46.580 INFO:teuthology.kill:No teuthology processes running
2021-05-19T10:55:47.676 INFO:teuthology.kill:Nuking machines: ['smithi105', 'smithi114']
2021-05-19T10:55:48.616 INFO:teuthology.kill:2021-05-19 10:55:48,615.615 INFO:teuthology.nuke:targets:
2021-05-19T10:55:48.617 INFO:teuthology.kill: smithi105.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8bbfbLIUHGzwdk+eFknNLqJ2a0d6TEtaqJegPU6S7zPAGSA3ONEhR9UDIGz4RiS2ci7dhyXjZzPQAEQxnvsW6vPLlR/hg3ssEaPuxYcj5gWyjQ6ziVNeLtDKjcC5laAajd+J7Ic0J3HZEjmtDQf9PUNv/w7FAK/7kP5DDcU5cOUqrxpmK/AB7ONmh2GehBgP4tYB8A6x81+Zik56HvQ2EsaOdjDrG51g/YdRPtnmt/yte8tQG0nEHZ4FUWTAwXVCHycm1PRVyY1rpeoCSBQY3cFnv90WpZtPtwfC8QC+bJkfM6y7m5spGQt966dGb1/8o1l2I/ICAru92zFOIWnD4VGmvT2K4ZvRXNbKZo4rn4U83qgrS6Foy1QxRP8oMkOzXURvzNPs/7f/y4pq27BX/+a+5cva6GfdkH4fu9oUZlQxuPkat/rEle3O0+CoyEDT/HVg4f6zSQI5tZfyYgdxVHBo7OrO7EfwRc6WmtcXFt5HFC4sJoayqMViucSZdt6c=
2021-05-19T10:55:48.618 INFO:teuthology.kill: smithi114.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCcf20IWWVnhlPtoP/CHmRfsJrBWUFQU2f/KtHE6GofZuqj3GxN/mN5yFtQmvpEfNWK59Ze2vY8nV+ObeUPw+zAw7YFIimasq2oBebaVIi7bGouGb7QksajEgy1OUUivddzYrxrJOFebzBZgqAE5lwokKMq/PtAxPh4ccj++DXKZOYo/l97aWvybYX8XcWXy3XF5xUXjHRLNJ4bGuu0xtxJyKcs6wcfw6QlMFDIdJ1cnwYwjLZki0nj7wZdBvXhDgTiuu/FZjzOK1YxqmcrM+paFHZfgFtnB8iFdxJiPkAGeSpJB2Ll9tCsxYqxdf2OUgm+ww7nt420AzbFy8q5odePDFppRr2wSKE8LVQ20y1+tgjgJLvjjiu5yDZnBVDTLl3TNbHgMOihfeShEIdZumvEPUZUU67KEmX+L+MBkmAgkOE4yq0jb+FylU71mSkyPHGmutLtaaX+558dcH2xfW0vZvxGp08asgP/MXd1by0BhEDokhm4L1LpHgzledF2JTU=
2021-05-19T10:55:48.636 INFO:teuthology.kill:2021-05-19 10:55:48,635.635 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-19T10:55:48.640 INFO:teuthology.kill:2021-05-19 10:55:48,640.640 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-19T10:55:48.684 INFO:teuthology.kill:2021-05-19 10:55:48,683.683 INFO:teuthology.orchestra.console:Power off smithi105
2021-05-19T10:55:48.697 INFO:teuthology.kill:2021-05-19 10:55:48,697.697 INFO:teuthology.orchestra.console:Power off smithi114
2021-05-19T10:55:48.824 INFO:teuthology.kill:2021-05-19 10:55:48,824.824 INFO:teuthology.orchestra.console:Power off for smithi105 completed
2021-05-19T10:55:48.835 INFO:teuthology.kill:2021-05-19 10:55:48,835.835 INFO:teuthology.orchestra.console:Power off for smithi114 completed
2021-05-19T10:55:48.961 INFO:teuthology.kill:2021-05-19 10:55:48,961.961 INFO:teuthology.lock.ops:unlocked smithi105.front.sepia.ceph.com
2021-05-19T10:55:48.970 INFO:teuthology.kill:2021-05-19 10:55:48,970.970 INFO:teuthology.lock.ops:unlocked smithi114.front.sepia.ceph.com
2021-05-19T10:57:49.280 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-05-19T10:57:49.361 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2021-05-19T10:57:49.362 INFO:teuthology.nuke:Checking targets against current locks
2021-05-19T10:57:49.465 INFO:teuthology.nuke:Not nuking smithi105.front.sepia.ceph.com because description doesn't match
2021-05-19T10:57:49.465 INFO:teuthology.nuke:Not nuking smithi114.front.sepia.ceph.com because description doesn't match