2021-04-29T01:31:37.004 INFO:root:teuthology version: 1.1.0-2713a3cd
2021-04-29T01:31:37.109 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba023.front.sepia.ceph.com' reimaging
2021-04-29T01:31:37.110 INFO:teuthology.lock.ops:Updating [ubuntu@gibba023.front.sepia.ceph.com]: reset os type and version on server
2021-04-29T01:31:37.110 INFO:teuthology.lock.ops:Updating gibba023.front.sepia.ceph.com on lock server
2021-04-29T01:31:37.125 INFO:teuthology.lock.ops:Node 'ubuntu@gibba023.front.sepia.ceph.com' reimaging is complete
2021-04-29T01:31:37.125 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba026.front.sepia.ceph.com' reimaging
2021-04-29T01:31:37.125 INFO:teuthology.lock.ops:Updating [ubuntu@gibba026.front.sepia.ceph.com]: reset os type and version on server
2021-04-29T01:31:37.125 INFO:teuthology.lock.ops:Updating gibba026.front.sepia.ceph.com on lock server
2021-04-29T01:31:37.139 INFO:teuthology.lock.ops:Node 'ubuntu@gibba026.front.sepia.ceph.com' reimaging is complete
2021-04-29T01:31:37.318 INFO:teuthology.provision.fog.gibba023:Scheduling deploy of ubuntu 20.04
2021-04-29T01:31:37.330 INFO:teuthology.provision.fog.gibba026:Scheduling deploy of ubuntu 20.04
2021-04-29T01:31:37.609 INFO:teuthology.orchestra.console:Power off gibba023
2021-04-29T01:31:37.621 INFO:teuthology.orchestra.console:Power off gibba026
2021-04-29T01:31:45.984 INFO:teuthology.orchestra.console:Power off for gibba023 completed
2021-04-29T01:31:45.998 INFO:teuthology.orchestra.console:Power off for gibba026 completed
2021-04-29T01:31:46.086 INFO:teuthology.orchestra.console:Power on gibba023
2021-04-29T01:31:46.099 INFO:teuthology.orchestra.console:Power on gibba026
2021-04-29T01:31:54.462 INFO:teuthology.orchestra.console:Power on for gibba023 completed
2021-04-29T01:31:54.468 INFO:teuthology.orchestra.console:Power on for gibba026 completed
2021-04-29T01:31:54.563 INFO:teuthology.provision.fog.gibba023:Waiting for deploy to finish
2021-04-29T01:31:54.568 INFO:teuthology.provision.fog.gibba026:Waiting for deploy to finish
2021-04-29T01:36:34.399 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-04-29T01:36:34.898 INFO:teuthology.provision.fog.gibba026:Node is ready
2021-04-29T01:36:34.949 INFO:teuthology.orchestra.run.gibba026.stdout:gibba026.front.sepia.ceph.com
2021-04-29T01:36:34.958 INFO:teuthology.orchestra.run.gibba026.stdout:172.21.2.126 gibba026.front.sepia.ceph.com gibba026
2021-04-29T01:36:35.112 INFO:teuthology.provision.fog.gibba026:Deploy complete!
2021-04-29T01:36:36.940 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-04-29T01:36:37.400 INFO:teuthology.provision.fog.gibba023:Node is ready
2021-04-29T01:36:37.407 INFO:teuthology.orchestra.run.gibba023.stdout:gibba023.front.sepia.ceph.com
2021-04-29T01:36:37.414 INFO:teuthology.orchestra.run.gibba023.stdout:172.21.2.123 gibba023.front.sepia.ceph.com gibba023
2021-04-29T01:36:37.568 INFO:teuthology.provision.fog.gibba023:Deploy complete!
2021-04-29T01:37:03.691 INFO:teuthology.lock.ops:Checking gibba023.front.sepia.ceph.com
2021-04-29T01:37:03.692 INFO:teuthology.lock.ops:New key found. Updating...
2021-04-29T01:37:06.288 INFO:teuthology.lock.ops:Checking gibba026.front.sepia.ceph.com
2021-04-29T01:37:06.288 INFO:teuthology.lock.ops:New key found. Updating...
2021-04-29T01:37:06.353 INFO:teuthology.lock.ops:Updating [gibba023.front.sepia.ceph.com]: set os type and version on server
2021-04-29T01:37:06.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-04-29T01:37:06.523 INFO:teuthology.orchestra.run.gibba023.stdout:x86_64
2021-04-29T01:37:06.574 INFO:teuthology.orchestra.run.gibba023.stdout:NAME="Ubuntu"
2021-04-29T01:37:06.575 INFO:teuthology.orchestra.run.gibba023.stdout:VERSION="20.04.2 LTS (Focal Fossa)"
2021-04-29T01:37:06.575 INFO:teuthology.orchestra.run.gibba023.stdout:ID=ubuntu
2021-04-29T01:37:06.575 INFO:teuthology.orchestra.run.gibba023.stdout:ID_LIKE=debian
2021-04-29T01:37:06.575 INFO:teuthology.orchestra.run.gibba023.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS"
2021-04-29T01:37:06.575 INFO:teuthology.orchestra.run.gibba023.stdout:VERSION_ID="20.04"
2021-04-29T01:37:06.575 INFO:teuthology.orchestra.run.gibba023.stdout:HOME_URL="https://www.ubuntu.com/"
2021-04-29T01:37:06.575 INFO:teuthology.orchestra.run.gibba023.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-04-29T01:37:06.575 INFO:teuthology.orchestra.run.gibba023.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-04-29T01:37:06.575 INFO:teuthology.orchestra.run.gibba023.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-04-29T01:37:06.575 INFO:teuthology.orchestra.run.gibba023.stdout:VERSION_CODENAME=focal
2021-04-29T01:37:06.576 INFO:teuthology.orchestra.run.gibba023.stdout:UBUNTU_CODENAME=focal
2021-04-29T01:37:06.576 INFO:teuthology.lock.ops:Updating gibba023.front.sepia.ceph.com on lock server
2021-04-29T01:37:06.594 INFO:teuthology.lock.ops:Updating [gibba026.front.sepia.ceph.com]: set os type and version on server
2021-04-29T01:37:06.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-04-29T01:37:06.727 INFO:teuthology.orchestra.run.gibba026.stdout:x86_64
2021-04-29T01:37:06.732 INFO:teuthology.orchestra.run.gibba026.stdout:NAME="Ubuntu"
2021-04-29T01:37:06.732 INFO:teuthology.orchestra.run.gibba026.stdout:VERSION="20.04.2 LTS (Focal Fossa)"
2021-04-29T01:37:06.732 INFO:teuthology.orchestra.run.gibba026.stdout:ID=ubuntu
2021-04-29T01:37:06.732 INFO:teuthology.orchestra.run.gibba026.stdout:ID_LIKE=debian
2021-04-29T01:37:06.732 INFO:teuthology.orchestra.run.gibba026.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS"
2021-04-29T01:37:06.733 INFO:teuthology.orchestra.run.gibba026.stdout:VERSION_ID="20.04"
2021-04-29T01:37:06.733 INFO:teuthology.orchestra.run.gibba026.stdout:HOME_URL="https://www.ubuntu.com/"
2021-04-29T01:37:06.733 INFO:teuthology.orchestra.run.gibba026.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-04-29T01:37:06.733 INFO:teuthology.orchestra.run.gibba026.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-04-29T01:37:06.733 INFO:teuthology.orchestra.run.gibba026.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-04-29T01:37:06.733 INFO:teuthology.orchestra.run.gibba026.stdout:VERSION_CODENAME=focal
2021-04-29T01:37:06.733 INFO:teuthology.orchestra.run.gibba026.stdout:UBUNTU_CODENAME=focal
2021-04-29T01:37:06.733 INFO:teuthology.lock.ops:Updating gibba026.front.sepia.ceph.com on lock server
2021-04-29T01:37:06.817 INFO:teuthology.dispatcher.supervisor:Running job 6056905
2021-04-29T01:37:06.817 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-04-19_02:01:03-rbd-master-distro-basic-gibba/6056905 --name teuthology-2021-04-19_02:01:03-rbd-master-distro-basic-gibba --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} -- /home/teuthworker/archive/teuthology-2021-04-19_02:01:03-rbd-master-distro-basic-gibba/6056905/orig.config.yaml
2021-04-29T01:37:06.820 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-04-19_02:01:03-rbd-master-distro-basic-gibba/6056905
2021-04-29T01:37:06.820 INFO:teuthology.dispatcher.supervisor:Job PID: 30836
2021-04-29T01:37:06.821 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-04-29T03:13:30.832 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/teuthology-2021-04-19_02:01:03-rbd-master-distro-basic-gibba/jobs/ failed with status 504:
504 Gateway Time-out
504 Gateway Time-out
nginx/1.10.3 (Ubuntu)
2021-04-29T13:39:05.664 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-04-29T13:39:05.929 INFO:teuthology.kill:Killing Pids: {30836}
2021-04-29T13:39:07.809 INFO:teuthology.kill:Nuking machines: ['gibba023', 'gibba026']
2021-04-29T13:39:08.768 INFO:teuthology.kill:2021-04-29 13:39:08,767.767 INFO:teuthology.nuke:targets:
2021-04-29T13:39:08.768 INFO:teuthology.kill: gibba023.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp6/vMJmmo42mHop3XQgGszLzs/re+LKNTxs/HvsOfedvnDaDtWoo8u+fBwhw0veHv7Be1NA5Eqm41n0QRwOdxjL/nyje/eJfCm/KVOQSJi7b+zz1XlMQuLXNW4JyKOcvfjHXnP7NJBu6qPbonsF3ueCwlVqUk9DAuNzYvNpCyDcaXkAYTEVC2HsbD2VdIPIOJBUChU1qnFIhBEzJ3p4m6GjYk1BLN8gGLxYh5v+Qx4dcrgtY6VkO5z88qtSy64zjbCWj6F95ZpEDfrAuEDz45ttw15KXi6+N29flVE70MHIGGrwl45qfq0LUs/zP3CmOwJgN42RpYuDW0F07fiP36bpQNMpb/2UWKRyRFXfzJ30Sft3ahYhI1EvaAbMUu0UE93w5s/23w49g2graMLaM9laOSebPLm4glBph7ibRGPu5ktS4/L/ouV4D2Eu7iXFTxc9k3pod4DTTGb7BdBT+f8IZnD5kTSNfJkjKejGcxt68LLPIkDBDy6MIFukEA4ME=
2021-04-29T13:39:08.768 INFO:teuthology.kill: gibba026.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCylU4ZGeakabMo9/RclIi3jVQpwiQY+/T/3PzruIhyd8EcF9wUrsfArkSGgIRFcRsDAbSBqxQnIABTLXYDDUt+jD2YrApGMcvBRMtj8LEZ0GKHPM2+usAo4kKJ9OE6ciGsDGrhz/qoaIo3nMhT71rRMVrEd/raaGjRvcQCQrlcyoKDzlGynuogaBQlcTRwuPbvA9rEXzIjVntG91fx4ssLfxB6p33WgPCjizvNr/k6E/aouMUNcEkEOgTSpkg0jrPCHJWxnDf6CRM6NF9JCoPP8ArjxuEGavXPDB0PGoORLpcXZaPhujimBzdAOR8KOMjEjkQOH0c9wTdjJ6jBZkhXRUHuFWZTyyasD158doNDmQw5PaMEmKSeOLFnOvSMloKr3XRNhCYsWnxlbR6BId8WQzyQdrgODqxMtEpM1Du/EM+QHKxmooAsfx8VRjm0OhgA3LS9g3hVFVPzWUQ/mgNB/TWChcV4/Nca5SLehid9fwN3Z+uKABrEYyXEyl7A/Oc=
2021-04-29T13:39:08.768 INFO:teuthology.kill:2021-04-29 13:39:08,768.768 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-29T13:39:08.770 INFO:teuthology.kill:2021-04-29 13:39:08,770.770 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-29T13:39:08.842 INFO:teuthology.kill:2021-04-29 13:39:08,841.841 INFO:teuthology.orchestra.console:Power off gibba023
2021-04-29T13:39:08.855 INFO:teuthology.kill:2021-04-29 13:39:08,854.854 INFO:teuthology.orchestra.console:Power off gibba026
2021-04-29T13:39:17.326 INFO:teuthology.kill:2021-04-29 13:39:17,326.326 INFO:teuthology.orchestra.console:Power off for gibba026 completed
2021-04-29T13:39:21.328 INFO:teuthology.kill:2021-04-29 13:39:21,327.327 INFO:teuthology.orchestra.console:Power off for gibba023 completed
2021-04-29T13:39:21.593 INFO:teuthology.task.internal:roles: ubuntu@gibba023.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0']
2021-04-29T13:39:21.593 INFO:teuthology.task.internal:roles: ubuntu@gibba026.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']
2021-04-29T13:39:21.593 INFO:teuthology.misc:Compressing logs...
2021-04-29T13:39:21.593 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-04-29T13:39:39.995 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
teuth_config.archive_base, job_config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
compress_logs(ctx, log_path)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/misc.py", line 1378, in compress_logs
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/remote.py", line 215, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/run.py", line 434, in run
transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-04-29T13:39:40.068 INFO:teuthology.kill:No teuthology processes running
2021-04-29T13:39:41.132 INFO:teuthology.kill:Nuking machines: ['gibba023', 'gibba026']
2021-04-29T13:39:42.069 INFO:teuthology.kill:2021-04-29 13:39:42,068.068 INFO:teuthology.nuke:targets:
2021-04-29T13:39:42.069 INFO:teuthology.kill: gibba023.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp6/vMJmmo42mHop3XQgGszLzs/re+LKNTxs/HvsOfedvnDaDtWoo8u+fBwhw0veHv7Be1NA5Eqm41n0QRwOdxjL/nyje/eJfCm/KVOQSJi7b+zz1XlMQuLXNW4JyKOcvfjHXnP7NJBu6qPbonsF3ueCwlVqUk9DAuNzYvNpCyDcaXkAYTEVC2HsbD2VdIPIOJBUChU1qnFIhBEzJ3p4m6GjYk1BLN8gGLxYh5v+Qx4dcrgtY6VkO5z88qtSy64zjbCWj6F95ZpEDfrAuEDz45ttw15KXi6+N29flVE70MHIGGrwl45qfq0LUs/zP3CmOwJgN42RpYuDW0F07fiP36bpQNMpb/2UWKRyRFXfzJ30Sft3ahYhI1EvaAbMUu0UE93w5s/23w49g2graMLaM9laOSebPLm4glBph7ibRGPu5ktS4/L/ouV4D2Eu7iXFTxc9k3pod4DTTGb7BdBT+f8IZnD5kTSNfJkjKejGcxt68LLPIkDBDy6MIFukEA4ME=
2021-04-29T13:39:42.069 INFO:teuthology.kill: gibba026.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCylU4ZGeakabMo9/RclIi3jVQpwiQY+/T/3PzruIhyd8EcF9wUrsfArkSGgIRFcRsDAbSBqxQnIABTLXYDDUt+jD2YrApGMcvBRMtj8LEZ0GKHPM2+usAo4kKJ9OE6ciGsDGrhz/qoaIo3nMhT71rRMVrEd/raaGjRvcQCQrlcyoKDzlGynuogaBQlcTRwuPbvA9rEXzIjVntG91fx4ssLfxB6p33WgPCjizvNr/k6E/aouMUNcEkEOgTSpkg0jrPCHJWxnDf6CRM6NF9JCoPP8ArjxuEGavXPDB0PGoORLpcXZaPhujimBzdAOR8KOMjEjkQOH0c9wTdjJ6jBZkhXRUHuFWZTyyasD158doNDmQw5PaMEmKSeOLFnOvSMloKr3XRNhCYsWnxlbR6BId8WQzyQdrgODqxMtEpM1Du/EM+QHKxmooAsfx8VRjm0OhgA3LS9g3hVFVPzWUQ/mgNB/TWChcV4/Nca5SLehid9fwN3Z+uKABrEYyXEyl7A/Oc=
2021-04-29T13:39:42.091 INFO:teuthology.kill:2021-04-29 13:39:42,091.091 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-29T13:39:42.105 INFO:teuthology.kill:2021-04-29 13:39:42,105.105 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-29T13:39:42.177 INFO:teuthology.kill:2021-04-29 13:39:42,177.177 INFO:teuthology.orchestra.console:Power off gibba023
2021-04-29T13:39:42.191 INFO:teuthology.kill:2021-04-29 13:39:42,191.191 INFO:teuthology.orchestra.console:Power off gibba026
2021-04-29T13:39:42.325 INFO:teuthology.kill:2021-04-29 13:39:42,325.325 INFO:teuthology.orchestra.console:Power off for gibba023 completed
2021-04-29T13:39:42.342 INFO:teuthology.kill:2021-04-29 13:39:42,342.342 INFO:teuthology.orchestra.console:Power off for gibba026 completed
2021-04-29T13:39:42.483 INFO:teuthology.kill:2021-04-29 13:39:42,482.482 INFO:teuthology.lock.ops:unlocked gibba023.front.sepia.ceph.com
2021-04-29T13:39:42.503 INFO:teuthology.kill:2021-04-29 13:39:42,503.503 INFO:teuthology.lock.ops:unlocked gibba026.front.sepia.ceph.com
2021-04-29T13:41:42.801 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15