2021-05-27T15:28:11.097 INFO:root:teuthology version: 1.1.0-630d5963
2021-05-27T15:28:11.189 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba033.front.sepia.ceph.com' reimaging
2021-05-27T15:28:11.191 INFO:teuthology.lock.ops:Updating [ubuntu@gibba033.front.sepia.ceph.com]: reset os type and version on server
2021-05-27T15:28:11.192 INFO:teuthology.lock.ops:Updating gibba033.front.sepia.ceph.com on lock server
2021-05-27T15:28:11.208 INFO:teuthology.lock.ops:Node 'ubuntu@gibba033.front.sepia.ceph.com' reimaging is complete
2021-05-27T15:28:11.209 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba041.front.sepia.ceph.com' reimaging
2021-05-27T15:28:11.209 INFO:teuthology.lock.ops:Updating [ubuntu@gibba041.front.sepia.ceph.com]: reset os type and version on server
2021-05-27T15:28:11.210 INFO:teuthology.lock.ops:Updating gibba041.front.sepia.ceph.com on lock server
2021-05-27T15:28:11.227 INFO:teuthology.lock.ops:Node 'ubuntu@gibba041.front.sepia.ceph.com' reimaging is complete
2021-05-27T15:28:11.439 INFO:teuthology.provision.fog.gibba033:Scheduling deploy of ubuntu 20.04
2021-05-27T15:28:11.453 INFO:teuthology.provision.fog.gibba041:Scheduling deploy of ubuntu 20.04
2021-05-27T15:28:11.806 INFO:teuthology.orchestra.console:Power off gibba033
2021-05-27T15:28:11.849 INFO:teuthology.orchestra.console:Power off gibba041
2021-05-27T15:28:20.186 INFO:teuthology.orchestra.console:Power off for gibba033 completed
2021-05-27T15:28:20.220 INFO:teuthology.orchestra.console:Power off for gibba041 completed
2021-05-27T15:28:20.288 INFO:teuthology.orchestra.console:Power on gibba033
2021-05-27T15:28:20.321 INFO:teuthology.orchestra.console:Power on gibba041
2021-05-27T15:28:28.665 INFO:teuthology.orchestra.console:Power on for gibba033 completed
2021-05-27T15:28:28.689 INFO:teuthology.orchestra.console:Power on for gibba041 completed
2021-05-27T15:28:28.767 INFO:teuthology.provision.fog.gibba033:Waiting for deploy to finish
2021-05-27T15:28:28.790 INFO:teuthology.provision.fog.gibba041:Waiting for deploy to finish
2021-05-27T15:33:08.547 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-05-27T15:33:08.844 INFO:teuthology.provision.fog.gibba033:Node is ready
2021-05-27T15:33:08.849 INFO:teuthology.orchestra.run.gibba033.stdout:gibba033.front.sepia.ceph.com
2021-05-27T15:33:08.854 INFO:teuthology.orchestra.run.gibba033.stdout:172.21.2.133 gibba033.front.sepia.ceph.com gibba033
2021-05-27T15:33:08.955 INFO:teuthology.provision.fog.gibba033:Deploy complete!
2021-05-27T15:33:19.018 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-05-27T15:33:19.503 INFO:teuthology.provision.fog.gibba041:Node is ready
2021-05-27T15:33:19.510 INFO:teuthology.orchestra.run.gibba041.stdout:gibba041.front.sepia.ceph.com
2021-05-27T15:33:19.520 INFO:teuthology.orchestra.run.gibba041.stdout:172.21.2.141 gibba041.front.sepia.ceph.com gibba041
2021-05-27T15:33:19.656 INFO:teuthology.provision.fog.gibba041:Deploy complete!
2021-05-27T15:33:19.786 INFO:teuthology.lock.ops:Checking gibba033.front.sepia.ceph.com
2021-05-27T15:33:19.787 INFO:teuthology.lock.ops:New key found. Updating...
2021-05-27T15:33:19.806 INFO:teuthology.lock.ops:Checking gibba041.front.sepia.ceph.com
2021-05-27T15:33:19.806 INFO:teuthology.lock.ops:New key found. Updating...
2021-05-27T15:33:19.821 INFO:teuthology.lock.ops:Updating [gibba033.front.sepia.ceph.com]: set os type and version on server
2021-05-27T15:33:19.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-05-27T15:33:19.953 INFO:teuthology.orchestra.run.gibba033.stdout:x86_64
2021-05-27T15:33:20.001 INFO:teuthology.orchestra.run.gibba033.stdout:NAME="Ubuntu"
2021-05-27T15:33:20.001 INFO:teuthology.orchestra.run.gibba033.stdout:VERSION="20.04.2 LTS (Focal Fossa)"
2021-05-27T15:33:20.001 INFO:teuthology.orchestra.run.gibba033.stdout:ID=ubuntu
2021-05-27T15:33:20.002 INFO:teuthology.orchestra.run.gibba033.stdout:ID_LIKE=debian
2021-05-27T15:33:20.002 INFO:teuthology.orchestra.run.gibba033.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS"
2021-05-27T15:33:20.002 INFO:teuthology.orchestra.run.gibba033.stdout:VERSION_ID="20.04"
2021-05-27T15:33:20.003 INFO:teuthology.orchestra.run.gibba033.stdout:HOME_URL="https://www.ubuntu.com/"
2021-05-27T15:33:20.003 INFO:teuthology.orchestra.run.gibba033.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-05-27T15:33:20.003 INFO:teuthology.orchestra.run.gibba033.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-05-27T15:33:20.003 INFO:teuthology.orchestra.run.gibba033.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-05-27T15:33:20.003 INFO:teuthology.orchestra.run.gibba033.stdout:VERSION_CODENAME=focal
2021-05-27T15:33:20.004 INFO:teuthology.orchestra.run.gibba033.stdout:UBUNTU_CODENAME=focal
2021-05-27T15:33:20.005 INFO:teuthology.lock.ops:Updating gibba033.front.sepia.ceph.com on lock server
2021-05-27T15:33:20.019 INFO:teuthology.lock.ops:Updating [gibba041.front.sepia.ceph.com]: set os type and version on server
2021-05-27T15:33:20.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-05-27T15:33:20.209 INFO:teuthology.orchestra.run.gibba041.stdout:x86_64
2021-05-27T15:33:20.217 INFO:teuthology.orchestra.run.gibba041.stdout:NAME="Ubuntu"
2021-05-27T15:33:20.218 INFO:teuthology.orchestra.run.gibba041.stdout:VERSION="20.04.2 LTS (Focal Fossa)"
2021-05-27T15:33:20.218 INFO:teuthology.orchestra.run.gibba041.stdout:ID=ubuntu
2021-05-27T15:33:20.218 INFO:teuthology.orchestra.run.gibba041.stdout:ID_LIKE=debian
2021-05-27T15:33:20.218 INFO:teuthology.orchestra.run.gibba041.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS"
2021-05-27T15:33:20.219 INFO:teuthology.orchestra.run.gibba041.stdout:VERSION_ID="20.04"
2021-05-27T15:33:20.219 INFO:teuthology.orchestra.run.gibba041.stdout:HOME_URL="https://www.ubuntu.com/"
2021-05-27T15:33:20.219 INFO:teuthology.orchestra.run.gibba041.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-05-27T15:33:20.219 INFO:teuthology.orchestra.run.gibba041.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-05-27T15:33:20.219 INFO:teuthology.orchestra.run.gibba041.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-05-27T15:33:20.220 INFO:teuthology.orchestra.run.gibba041.stdout:VERSION_CODENAME=focal
2021-05-27T15:33:20.220 INFO:teuthology.orchestra.run.gibba041.stdout:UBUNTU_CODENAME=focal
2021-05-27T15:33:20.220 INFO:teuthology.lock.ops:Updating gibba041.front.sepia.ceph.com on lock server
2021-05-27T15:33:20.309 INFO:teuthology.dispatcher.supervisor:Running job 6118959
2021-05-27T15:33:20.310 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_630d5963a6194f6b5a287f9a69bf6cea3ae44211/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-05-18_03:15:03-fs-master-distro-basic-gibba/6118959 --name teuthology-2021-05-18_03:15:03-fs-master-distro-basic-gibba --description fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc} -- /home/teuthworker/archive/teuthology-2021-05-18_03:15:03-fs-master-distro-basic-gibba/6118959/orig.config.yaml
2021-05-27T15:33:20.314 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-05-18_03:15:03-fs-master-distro-basic-gibba/6118959
2021-05-27T15:33:20.315 INFO:teuthology.dispatcher.supervisor:Job PID: 23691
2021-05-27T15:33:20.315 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-05-27T18:17:31.975 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/teuthology-2021-05-18_03:15:03-fs-master-distro-basic-gibba/jobs/ failed with status 500:
Internal Server Error
Internal Server Error
2021-05-28T03:34:12.780 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-05-28T03:34:12.991 INFO:teuthology.kill:Killing Pids: {23691}
2021-05-28T03:34:14.137 INFO:teuthology.kill:Nuking machines: ['gibba033', 'gibba041']
2021-05-28T03:34:15.078 INFO:teuthology.kill:2021-05-28 03:34:15,077.077 INFO:teuthology.nuke:targets:
2021-05-28T03:34:15.079 INFO:teuthology.kill: gibba033.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZ9VtOu02fV30MMJBD3oobygSE5ByPhruD6xGP8l0AzOAfYMaW98Q0OjPH42BE5k968gs6eGPsk4AX10ECiAIhRwWukOetlN58g3595GB8lAsAl2KVDSExcXWhaA38mOpRYfMpa/YsGEk6bFgDkxEDvIYfZUTFisR/KtEy0iOG4nWHo1o3zpEP138uvKrupRbVD5VvWc4D2HFc0m9f5KxotsN8kPJXVIZWRHfR8fV1zkh5qleaok91RASUO83X2+QOYJ1r/hbZjYgdlqGU0vMQVDT7n9Wm8930mIu5d3/VvFGZROGUT9Yb3cDvSAehV5YKWTsYhsQ6nr8ulhoqvrAdWguJbWjALP+gkjllh348G9O3bu27zQVu+/16FzL9J7h6mRJ28tYFvJ1YedK+FC70NWl0ulj3hYagaosNaMGfIM1AinscZDp0F1jwDTAbBz1J+Hivr2lwO/ocGymEYN6I+tx9PHsU4kWJkZYU88jdxWrDVpo0GmcnfkM475a3Wc8=
2021-05-28T03:34:15.080 INFO:teuthology.kill: gibba041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2fjYHbsLurVIrosfcskLMLF2ZA6oLRv6UdpWfFrvEUaXHOqjEHIRqFWRBQpV6ExV5MJnZ1xN9xz/i3NE0hvPYax3WYDdmCRgBtT5+YkQi9vJtnmM5xOeXh4zK8q28JFOS1+ZrlWTiUHzfPXLg/IWpQ0GwjxB2oMRbSRboZeYzY1YQBCJ39Yefn8XTV+iu1czWdKjzxpuMSqRglEo4o0cXuCZt5PKvRXy+sc7EZuWHcJ1G+QZANqFNeaXA53/0OfWU7VRUisa1CCyG1l427khg8BtuHU4J/cSgxpJqKhdwuqYd0p4ZPuuOXJicaY9428AGXrJ5JLqxTqeyHw0ZKT5LoRGCslBt2EGuZUcoGaf5BpXPRiu+qMdPpz1g6gcQCFr+2nCh8biuP91h1N6SWDtb4dueY3D/Ix7QyNYdINAHcFRq6nmDq9/ZhxSZfIzBEgRPn7A32bMltYfP2IvoXFRyNg05WNsX1d71oyU7Z3uPY2PQWMPnF5k3GA4+SwF/Vgs=
2021-05-28T03:34:15.080 INFO:teuthology.kill:2021-05-28 03:34:15,078.078 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-28T03:34:15.080 INFO:teuthology.kill:2021-05-28 03:34:15,080.080 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-28T03:34:15.146 INFO:teuthology.kill:2021-05-28 03:34:15,146.146 INFO:teuthology.orchestra.console:Power off gibba033
2021-05-28T03:34:15.172 INFO:teuthology.kill:2021-05-28 03:34:15,171.171 INFO:teuthology.orchestra.console:Power off gibba041
2021-05-28T03:34:27.646 INFO:teuthology.kill:2021-05-28 03:34:27,645.645 INFO:teuthology.orchestra.console:Power off for gibba033 completed
2021-05-28T03:34:27.737 INFO:teuthology.kill:2021-05-28 03:34:27,737.737 INFO:teuthology.orchestra.console:Power off for gibba041 completed
2021-05-28T03:34:27.977 INFO:teuthology.task.internal:roles: ubuntu@gibba033.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2']
2021-05-28T03:34:27.977 INFO:teuthology.task.internal:roles: ubuntu@gibba041.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']
2021-05-28T03:34:27.978 INFO:teuthology.misc:Compressing logs...
2021-05-28T03:34:27.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-05-28T03:35:02.603 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_630d5963a6194f6b5a287f9a69bf6cea3ae44211/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
teuth_config.archive_base, job_config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_630d5963a6194f6b5a287f9a69bf6cea3ae44211/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
compress_logs(ctx, log_path)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_630d5963a6194f6b5a287f9a69bf6cea3ae44211/teuthology/misc.py", line 1378, in compress_logs
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_630d5963a6194f6b5a287f9a69bf6cea3ae44211/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_630d5963a6194f6b5a287f9a69bf6cea3ae44211/teuthology/orchestra/cluster.py", line 64, in
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_630d5963a6194f6b5a287f9a69bf6cea3ae44211/teuthology/orchestra/remote.py", line 509, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_630d5963a6194f6b5a287f9a69bf6cea3ae44211/teuthology/orchestra/run.py", line 434, in run
transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-05-28T03:35:02.668 INFO:teuthology.kill:No teuthology processes running
2021-05-28T03:35:03.771 INFO:teuthology.kill:Nuking machines: ['gibba033', 'gibba041']
2021-05-28T03:35:04.741 INFO:teuthology.kill:2021-05-28 03:35:04,741.741 INFO:teuthology.nuke:targets:
2021-05-28T03:35:04.742 INFO:teuthology.kill: gibba033.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZ9VtOu02fV30MMJBD3oobygSE5ByPhruD6xGP8l0AzOAfYMaW98Q0OjPH42BE5k968gs6eGPsk4AX10ECiAIhRwWukOetlN58g3595GB8lAsAl2KVDSExcXWhaA38mOpRYfMpa/YsGEk6bFgDkxEDvIYfZUTFisR/KtEy0iOG4nWHo1o3zpEP138uvKrupRbVD5VvWc4D2HFc0m9f5KxotsN8kPJXVIZWRHfR8fV1zkh5qleaok91RASUO83X2+QOYJ1r/hbZjYgdlqGU0vMQVDT7n9Wm8930mIu5d3/VvFGZROGUT9Yb3cDvSAehV5YKWTsYhsQ6nr8ulhoqvrAdWguJbWjALP+gkjllh348G9O3bu27zQVu+/16FzL9J7h6mRJ28tYFvJ1YedK+FC70NWl0ulj3hYagaosNaMGfIM1AinscZDp0F1jwDTAbBz1J+Hivr2lwO/ocGymEYN6I+tx9PHsU4kWJkZYU88jdxWrDVpo0GmcnfkM475a3Wc8=
2021-05-28T03:35:04.742 INFO:teuthology.kill: gibba041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2fjYHbsLurVIrosfcskLMLF2ZA6oLRv6UdpWfFrvEUaXHOqjEHIRqFWRBQpV6ExV5MJnZ1xN9xz/i3NE0hvPYax3WYDdmCRgBtT5+YkQi9vJtnmM5xOeXh4zK8q28JFOS1+ZrlWTiUHzfPXLg/IWpQ0GwjxB2oMRbSRboZeYzY1YQBCJ39Yefn8XTV+iu1czWdKjzxpuMSqRglEo4o0cXuCZt5PKvRXy+sc7EZuWHcJ1G+QZANqFNeaXA53/0OfWU7VRUisa1CCyG1l427khg8BtuHU4J/cSgxpJqKhdwuqYd0p4ZPuuOXJicaY9428AGXrJ5JLqxTqeyHw0ZKT5LoRGCslBt2EGuZUcoGaf5BpXPRiu+qMdPpz1g6gcQCFr+2nCh8biuP91h1N6SWDtb4dueY3D/Ix7QyNYdINAHcFRq6nmDq9/ZhxSZfIzBEgRPn7A32bMltYfP2IvoXFRyNg05WNsX1d71oyU7Z3uPY2PQWMPnF5k3GA4+SwF/Vgs=
2021-05-28T03:35:04.764 INFO:teuthology.kill:2021-05-28 03:35:04,764.764 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-28T03:35:04.773 INFO:teuthology.kill:2021-05-28 03:35:04,773.773 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-28T03:35:04.828 INFO:teuthology.kill:2021-05-28 03:35:04,828.828 INFO:teuthology.orchestra.console:Power off gibba033
2021-05-28T03:35:04.858 INFO:teuthology.kill:2021-05-28 03:35:04,858.858 INFO:teuthology.orchestra.console:Power off gibba041
2021-05-28T03:35:04.994 INFO:teuthology.kill:2021-05-28 03:35:04,994.994 INFO:teuthology.orchestra.console:Power off for gibba033 completed
2021-05-28T03:35:05.012 INFO:teuthology.kill:2021-05-28 03:35:05,011.011 INFO:teuthology.orchestra.console:Power off for gibba041 completed
2021-05-28T03:35:05.140 INFO:teuthology.kill:2021-05-28 03:35:05,138.138 INFO:teuthology.lock.ops:unlocked gibba033.front.sepia.ceph.com
2021-05-28T03:35:05.160 INFO:teuthology.kill:2021-05-28 03:35:05,160.160 INFO:teuthology.lock.ops:unlocked gibba041.front.sepia.ceph.com
2021-05-28T03:37:05.456 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-05-28T03:37:05.582 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2021-05-28T03:37:05.582 INFO:teuthology.nuke:Checking targets against current locks
2021-05-28T03:37:05.671 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-28T03:37:05.674 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-28T03:37:05.730 INFO:teuthology.orchestra.console:Power off gibba041
2021-05-28T03:37:05.744 INFO:teuthology.orchestra.console:Power off gibba033
2021-05-28T03:37:14.105 INFO:teuthology.orchestra.console:Power off for gibba041 completed
2021-05-28T03:37:14.253 INFO:teuthology.lock.ops:unlocked gibba041.front.sepia.ceph.com
2021-05-28T03:37:18.230 INFO:teuthology.orchestra.console:Power off for gibba033 completed
2021-05-28T03:37:18.367 ERROR:teuthology.lock.ops:failed to unlock gibba033.front.sepia.ceph.com. reason: 500