2021-08-11T11:04:44.736 INFO:root:teuthology version: 1.1.0-73aa7e3b 2021-08-11T11:04:44.835 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba004.front.sepia.ceph.com' reimaging 2021-08-11T11:04:44.836 INFO:teuthology.lock.ops:Updating [ubuntu@gibba004.front.sepia.ceph.com]: reset os type and version on server 2021-08-11T11:04:44.836 INFO:teuthology.lock.ops:Updating gibba004.front.sepia.ceph.com on lock server 2021-08-11T11:04:44.852 INFO:teuthology.lock.ops:Node 'ubuntu@gibba004.front.sepia.ceph.com' reimaging is complete 2021-08-11T11:04:44.853 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba017.front.sepia.ceph.com' reimaging 2021-08-11T11:04:44.853 INFO:teuthology.lock.ops:Updating [ubuntu@gibba017.front.sepia.ceph.com]: reset os type and version on server 2021-08-11T11:04:44.854 INFO:teuthology.lock.ops:Updating gibba017.front.sepia.ceph.com on lock server 2021-08-11T11:04:44.870 INFO:teuthology.lock.ops:Node 'ubuntu@gibba017.front.sepia.ceph.com' reimaging is complete 2021-08-11T11:04:45.053 INFO:teuthology.provision.fog.gibba004:Scheduling deploy of ubuntu 20.04 2021-08-11T11:04:45.057 INFO:teuthology.provision.fog.gibba017:Scheduling deploy of ubuntu 20.04 2021-08-11T11:04:45.366 INFO:teuthology.orchestra.console:Power off gibba017 2021-08-11T11:04:45.381 INFO:teuthology.orchestra.console:Power off gibba004 2021-08-11T11:04:53.744 INFO:teuthology.orchestra.console:Power off for gibba017 completed 2021-08-11T11:04:53.841 INFO:teuthology.orchestra.console:Power off for gibba004 completed 2021-08-11T11:04:53.846 INFO:teuthology.orchestra.console:Power on gibba017 2021-08-11T11:04:53.943 INFO:teuthology.orchestra.console:Power on gibba004 2021-08-11T11:05:02.216 INFO:teuthology.orchestra.console:Power on for gibba017 completed 2021-08-11T11:05:02.318 INFO:teuthology.provision.fog.gibba017:Waiting for deploy to finish 2021-08-11T11:05:02.330 INFO:teuthology.orchestra.console:Power on for gibba004 completed 2021-08-11T11:05:02.431 INFO:teuthology.provision.fog.gibba004:Waiting for deploy to finish 2021-08-11T11:09:42.667 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-11T11:09:43.065 INFO:teuthology.provision.fog.gibba017:Node is ready 2021-08-11T11:09:43.070 INFO:teuthology.orchestra.run.gibba017.stdout:gibba017.front.sepia.ceph.com 2021-08-11T11:09:43.076 INFO:teuthology.orchestra.run.gibba017.stdout:172.21.2.117 gibba017.front.sepia.ceph.com gibba017 2021-08-11T11:09:43.265 INFO:teuthology.provision.fog.gibba017:Deploy complete! 2021-08-11T11:09:44.670 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-11T11:09:45.099 INFO:teuthology.provision.fog.gibba004:Node is ready 2021-08-11T11:09:45.103 INFO:teuthology.orchestra.run.gibba004.stdout:gibba004.front.sepia.ceph.com 2021-08-11T11:09:45.153 INFO:teuthology.orchestra.run.gibba004.stdout:172.21.2.104 gibba004.front.sepia.ceph.com gibba004 2021-08-11T11:09:45.287 INFO:teuthology.provision.fog.gibba004:Deploy complete! 2021-08-11T11:09:45.427 INFO:teuthology.lock.ops:Checking gibba004.front.sepia.ceph.com 2021-08-11T11:09:45.428 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-11T11:09:45.447 INFO:teuthology.lock.ops:Checking gibba017.front.sepia.ceph.com 2021-08-11T11:09:45.448 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-11T11:09:45.465 INFO:teuthology.lock.ops:Updating [gibba004.front.sepia.ceph.com]: set os type and version on server 2021-08-11T11:09:45.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-11T11:09:45.636 INFO:teuthology.orchestra.run.gibba004.stdout:x86_64 2021-08-11T11:09:45.644 INFO:teuthology.orchestra.run.gibba004.stdout:NAME="Ubuntu" 2021-08-11T11:09:45.645 INFO:teuthology.orchestra.run.gibba004.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-08-11T11:09:45.645 INFO:teuthology.orchestra.run.gibba004.stdout:ID=ubuntu 2021-08-11T11:09:45.645 INFO:teuthology.orchestra.run.gibba004.stdout:ID_LIKE=debian 2021-08-11T11:09:45.645 INFO:teuthology.orchestra.run.gibba004.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-08-11T11:09:45.646 INFO:teuthology.orchestra.run.gibba004.stdout:VERSION_ID="20.04" 2021-08-11T11:09:45.646 INFO:teuthology.orchestra.run.gibba004.stdout:HOME_URL="https://www.ubuntu.com/" 2021-08-11T11:09:45.646 INFO:teuthology.orchestra.run.gibba004.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-08-11T11:09:45.646 INFO:teuthology.orchestra.run.gibba004.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-08-11T11:09:45.647 INFO:teuthology.orchestra.run.gibba004.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-08-11T11:09:45.647 INFO:teuthology.orchestra.run.gibba004.stdout:VERSION_CODENAME=focal 2021-08-11T11:09:45.647 INFO:teuthology.orchestra.run.gibba004.stdout:UBUNTU_CODENAME=focal 2021-08-11T11:09:45.648 INFO:teuthology.lock.ops:Updating gibba004.front.sepia.ceph.com on lock server 2021-08-11T11:09:45.664 INFO:teuthology.lock.ops:Updating [gibba017.front.sepia.ceph.com]: set os type and version on server 2021-08-11T11:09:45.665 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-11T11:09:45.797 INFO:teuthology.orchestra.run.gibba017.stdout:x86_64 2021-08-11T11:09:45.847 INFO:teuthology.orchestra.run.gibba017.stdout:NAME="Ubuntu" 2021-08-11T11:09:45.848 INFO:teuthology.orchestra.run.gibba017.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-08-11T11:09:45.848 INFO:teuthology.orchestra.run.gibba017.stdout:ID=ubuntu 2021-08-11T11:09:45.849 INFO:teuthology.orchestra.run.gibba017.stdout:ID_LIKE=debian 2021-08-11T11:09:45.849 INFO:teuthology.orchestra.run.gibba017.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-08-11T11:09:45.850 INFO:teuthology.orchestra.run.gibba017.stdout:VERSION_ID="20.04" 2021-08-11T11:09:45.850 INFO:teuthology.orchestra.run.gibba017.stdout:HOME_URL="https://www.ubuntu.com/" 2021-08-11T11:09:45.851 INFO:teuthology.orchestra.run.gibba017.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-08-11T11:09:45.851 INFO:teuthology.orchestra.run.gibba017.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-08-11T11:09:45.851 INFO:teuthology.orchestra.run.gibba017.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-08-11T11:09:45.852 INFO:teuthology.orchestra.run.gibba017.stdout:VERSION_CODENAME=focal 2021-08-11T11:09:45.852 INFO:teuthology.orchestra.run.gibba017.stdout:UBUNTU_CODENAME=focal 2021-08-11T11:09:45.853 INFO:teuthology.lock.ops:Updating gibba017.front.sepia.ceph.com on lock server 2021-08-11T11:09:45.937 INFO:teuthology.dispatcher.supervisor:Running job 6306655 2021-08-11T11:09:45.938 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-08-02_02:01:03-rbd-master-distro-basic-gibba/6306655 --name teuthology-2021-08-02_02:01:03-rbd-master-distro-basic-gibba --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} -- /home/teuthworker/archive/teuthology-2021-08-02_02:01:03-rbd-master-distro-basic-gibba/6306655/orig.config.yaml 2021-08-11T11:09:45.941 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-08-02_02:01:03-rbd-master-distro-basic-gibba/6306655 2021-08-11T11:09:45.942 INFO:teuthology.dispatcher.supervisor:Job PID: 16835 2021-08-11T11:09:45.942 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-08-11T23:10:35.999 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-08-11T23:10:36.306 INFO:teuthology.kill:Killing Pids: {16835} 2021-08-11T23:10:37.390 INFO:teuthology.kill:Nuking machines: ['gibba004', 'gibba017'] 2021-08-11T23:10:38.337 INFO:teuthology.kill:2021-08-11 23:10:38,336.336 INFO:teuthology.nuke:targets: 2021-08-11T23:10:38.337 INFO:teuthology.kill: gibba004.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkowhWKAYf+RL/nfCSDd2q9pOKtM+DPo2yIRXWzlsBNcrOTHz9QxsvJZBdlmpkK1eozvhPdSoLwWnHhxmUrmKNsBToER7ramJgXLYM4w66IqRemYVzjIQx/u/R9ixCmjlSn1okTWlKEYhuB5GpdAjPpsulve4/uL5ctFmTotxXLDoCLPCa1762ml3tsohBG2zjkyRK2OfmwdKlwwHjGp1zZ9TrSRpKnG5LZ4PE7U8BRyaTnmcJjV/RApCpio4gLyiS7Zr8kaZBPy+Io3zStCobqI/naTq16WSLimL5PeMjwlps6x44cv8PjHh83KMA/B/iPDCwqz8gZ02HEoLuWMtEFmrRm1eoBRaPKo1Yk5ZTR7At8n6affVuy9tviPbYI6pKhz2nC4MhNw4xlx5Mzfx2whB43U+pDKmi63Rq4TB6J2UtZSiPQbcG3whcQwOjK/tDN2y6sYlwi37UseiAau1P1qMBC66pNMZCUtTEl3ia1Q7Gz97x8QXGDDRsJm9oO0s= 2021-08-11T23:10:38.338 INFO:teuthology.kill: gibba017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCufnosXaMcOd4F2Jp+Q2JxYHZyfhpIcjPZQ/45Q305Looxi1hdz5jSZpDg7HN4gp1d5a741LA9NictI3AYg3hnYMpfCXCsU905sNLjgT81wjT6XISUJ0+vB23UPKs0kyGgxJIFoet3JWbg2lOQBDVCPhhEz6tLmyRAMdKpSBe0gcDrNPJV75Sskymj9vcJ/OKpq/yG/bTF2NO1hvdssbUmqRVeKN37ZYvvSTh8y2v82Qr0xbpY/jBsAArv+9qE/Bl2qn2PkM7q9efMMzBaMl7fq7UUqVccfqP/IyYLDGAgQjsY8o/8X+nJpu44lvvHghvyxGr378FGfNEga9NNSMOC7TNByV4PjaUvP2IWKCEcOjsZYlNSEgacEvjNptJ7Z/GJtQS58RjWId+HqWpEn6FubbATooDulX78KZgY0koGeCqAXtP0n4TR29e6ECOuQ7iw/bUBlksA0WlkSJleE2qVlO4rmouxc6JgRAT1vUcPpvXKdg1JkHyVfoLg+3EUE80= 2021-08-11T23:10:38.338 INFO:teuthology.kill:2021-08-11 23:10:38,337.337 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-11T23:10:38.339 INFO:teuthology.kill:2021-08-11 23:10:38,339.339 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-11T23:10:38.400 INFO:teuthology.kill:2021-08-11 23:10:38,400.400 INFO:teuthology.orchestra.console:Power off gibba004 2021-08-11T23:10:38.430 INFO:teuthology.kill:2021-08-11 23:10:38,430.430 INFO:teuthology.orchestra.console:Power off gibba017 2021-08-11T23:10:46.792 INFO:teuthology.kill:2021-08-11 23:10:46,791.791 INFO:teuthology.orchestra.console:Power off for gibba004 completed 2021-08-11T23:10:50.909 INFO:teuthology.kill:2021-08-11 23:10:50,908.908 INFO:teuthology.orchestra.console:Power off for gibba017 completed 2021-08-11T23:10:51.128 INFO:teuthology.task.internal:roles: ubuntu@gibba004.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2021-08-11T23:10:51.129 INFO:teuthology.task.internal:roles: ubuntu@gibba017.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2021-08-11T23:10:51.130 INFO:teuthology.misc:Compressing logs... 2021-08-11T23:10:51.130 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-11T23:11:09.515 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-08-11T23:11:09.579 INFO:teuthology.kill:No teuthology processes running 2021-08-11T23:11:10.640 INFO:teuthology.kill:Nuking machines: ['gibba004', 'gibba017'] 2021-08-11T23:11:11.566 INFO:teuthology.kill:2021-08-11 23:11:11,566.566 INFO:teuthology.nuke:targets: 2021-08-11T23:11:11.567 INFO:teuthology.kill: gibba004.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkowhWKAYf+RL/nfCSDd2q9pOKtM+DPo2yIRXWzlsBNcrOTHz9QxsvJZBdlmpkK1eozvhPdSoLwWnHhxmUrmKNsBToER7ramJgXLYM4w66IqRemYVzjIQx/u/R9ixCmjlSn1okTWlKEYhuB5GpdAjPpsulve4/uL5ctFmTotxXLDoCLPCa1762ml3tsohBG2zjkyRK2OfmwdKlwwHjGp1zZ9TrSRpKnG5LZ4PE7U8BRyaTnmcJjV/RApCpio4gLyiS7Zr8kaZBPy+Io3zStCobqI/naTq16WSLimL5PeMjwlps6x44cv8PjHh83KMA/B/iPDCwqz8gZ02HEoLuWMtEFmrRm1eoBRaPKo1Yk5ZTR7At8n6affVuy9tviPbYI6pKhz2nC4MhNw4xlx5Mzfx2whB43U+pDKmi63Rq4TB6J2UtZSiPQbcG3whcQwOjK/tDN2y6sYlwi37UseiAau1P1qMBC66pNMZCUtTEl3ia1Q7Gz97x8QXGDDRsJm9oO0s= 2021-08-11T23:11:11.568 INFO:teuthology.kill: gibba017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCufnosXaMcOd4F2Jp+Q2JxYHZyfhpIcjPZQ/45Q305Looxi1hdz5jSZpDg7HN4gp1d5a741LA9NictI3AYg3hnYMpfCXCsU905sNLjgT81wjT6XISUJ0+vB23UPKs0kyGgxJIFoet3JWbg2lOQBDVCPhhEz6tLmyRAMdKpSBe0gcDrNPJV75Sskymj9vcJ/OKpq/yG/bTF2NO1hvdssbUmqRVeKN37ZYvvSTh8y2v82Qr0xbpY/jBsAArv+9qE/Bl2qn2PkM7q9efMMzBaMl7fq7UUqVccfqP/IyYLDGAgQjsY8o/8X+nJpu44lvvHghvyxGr378FGfNEga9NNSMOC7TNByV4PjaUvP2IWKCEcOjsZYlNSEgacEvjNptJ7Z/GJtQS58RjWId+HqWpEn6FubbATooDulX78KZgY0koGeCqAXtP0n4TR29e6ECOuQ7iw/bUBlksA0WlkSJleE2qVlO4rmouxc6JgRAT1vUcPpvXKdg1JkHyVfoLg+3EUE80= 2021-08-11T23:11:11.586 INFO:teuthology.kill:2021-08-11 23:11:11,586.586 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-11T23:11:11.592 INFO:teuthology.kill:2021-08-11 23:11:11,592.592 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-11T23:11:11.649 INFO:teuthology.kill:2021-08-11 23:11:11,648.648 INFO:teuthology.orchestra.console:Power off gibba017 2021-08-11T23:11:11.662 INFO:teuthology.kill:2021-08-11 23:11:11,661.661 INFO:teuthology.orchestra.console:Power off gibba004 2021-08-11T23:11:11.801 INFO:teuthology.kill:2021-08-11 23:11:11,801.801 INFO:teuthology.orchestra.console:Power off for gibba017 completed 2021-08-11T23:11:11.808 INFO:teuthology.kill:2021-08-11 23:11:11,808.808 INFO:teuthology.orchestra.console:Power off for gibba004 completed 2021-08-11T23:11:11.943 INFO:teuthology.kill:2021-08-11 23:11:11,942.942 INFO:teuthology.lock.ops:unlocked gibba017.front.sepia.ceph.com 2021-08-11T23:11:11.946 INFO:teuthology.kill:2021-08-11 23:11:11,946.946 INFO:teuthology.lock.ops:unlocked gibba004.front.sepia.ceph.com 2021-08-11T23:13:12.242 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-08-11T23:13:12.346 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-08-11T23:13:12.347 INFO:teuthology.nuke:Checking targets against current locks 2021-08-11T23:13:12.465 INFO:teuthology.nuke:Not nuking gibba004.front.sepia.ceph.com because description doesn't match 2021-08-11T23:13:12.466 INFO:teuthology.nuke:Not nuking gibba017.front.sepia.ceph.com because description doesn't match