2021-03-05T02:46:46.482 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-05T02:46:46.595 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba017.front.sepia.ceph.com' reimaging 2021-03-05T02:46:46.597 INFO:teuthology.lock.ops:Updating [ubuntu@gibba017.front.sepia.ceph.com]: reset os type and version on server 2021-03-05T02:46:46.598 INFO:teuthology.lock.ops:Updating gibba017.front.sepia.ceph.com on lock server 2021-03-05T02:46:46.617 INFO:teuthology.lock.ops:Node 'ubuntu@gibba017.front.sepia.ceph.com' reimaging is complete 2021-03-05T02:46:46.618 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba039.front.sepia.ceph.com' reimaging 2021-03-05T02:46:46.618 INFO:teuthology.lock.ops:Updating [ubuntu@gibba039.front.sepia.ceph.com]: reset os type and version on server 2021-03-05T02:46:46.618 INFO:teuthology.lock.ops:Updating gibba039.front.sepia.ceph.com on lock server 2021-03-05T02:46:46.634 INFO:teuthology.lock.ops:Node 'ubuntu@gibba039.front.sepia.ceph.com' reimaging is complete 2021-03-05T02:46:46.829 INFO:teuthology.provision.fog.gibba017:Scheduling deploy of centos 8.2 2021-03-05T02:46:46.856 INFO:teuthology.provision.fog.gibba039:Scheduling deploy of centos 8.2 2021-03-05T02:46:47.154 INFO:teuthology.orchestra.console:Power off gibba017 2021-03-05T02:46:47.195 INFO:teuthology.orchestra.console:Power off gibba039 2021-03-05T02:46:55.531 INFO:teuthology.orchestra.console:Power off for gibba017 completed 2021-03-05T02:46:55.596 INFO:teuthology.orchestra.console:Power off for gibba039 completed 2021-03-05T02:46:55.633 INFO:teuthology.orchestra.console:Power on gibba017 2021-03-05T02:46:55.698 INFO:teuthology.orchestra.console:Power on gibba039 2021-03-05T02:47:04.012 INFO:teuthology.orchestra.console:Power on for gibba017 completed 2021-03-05T02:47:04.069 INFO:teuthology.orchestra.console:Power on for gibba039 completed 2021-03-05T02:47:04.114 INFO:teuthology.provision.fog.gibba017:Waiting for deploy to finish 2021-03-05T02:47:04.170 INFO:teuthology.provision.fog.gibba039:Waiting for deploy to finish 2021-03-05T02:51:04.656 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-05T02:51:14.898 INFO:teuthology.provision.fog.gibba039:Node is ready 2021-03-05T02:51:14.912 INFO:teuthology.orchestra.run.gibba039.stdout:gibba039.front.sepia.ceph.com 2021-03-05T02:51:14.965 INFO:teuthology.orchestra.run.gibba039.stdout:172.21.2.139 gibba039.front.sepia.ceph.com gibba039 2021-03-05T02:51:15.279 INFO:teuthology.provision.fog.gibba039:Deploy complete! 2021-03-05T02:51:30.075 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-05T02:51:30.302 INFO:teuthology.provision.fog.gibba017:Node is ready 2021-03-05T02:51:30.354 INFO:teuthology.orchestra.run.gibba017.stdout:gibba017.front.sepia.ceph.com 2021-03-05T02:51:30.407 INFO:teuthology.orchestra.run.gibba017.stdout:172.21.2.117 gibba017.front.sepia.ceph.com gibba017 2021-03-05T02:51:30.701 INFO:teuthology.provision.fog.gibba017:Deploy complete! 2021-03-05T02:51:30.814 INFO:teuthology.lock.ops:Checking gibba017.front.sepia.ceph.com 2021-03-05T02:51:30.815 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-05T02:51:30.833 INFO:teuthology.lock.ops:Checking gibba039.front.sepia.ceph.com 2021-03-05T02:51:30.833 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-05T02:51:30.861 INFO:teuthology.lock.ops:Updating [gibba017.front.sepia.ceph.com]: set os type and version on server 2021-03-05T02:51:30.862 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-05T02:51:31.094 INFO:teuthology.orchestra.run.gibba017.stdout:x86_64 2021-03-05T02:51:31.147 INFO:teuthology.orchestra.run.gibba017.stdout:NAME="CentOS Linux" 2021-03-05T02:51:31.147 INFO:teuthology.orchestra.run.gibba017.stdout:VERSION="8 (Core)" 2021-03-05T02:51:31.147 INFO:teuthology.orchestra.run.gibba017.stdout:ID="centos" 2021-03-05T02:51:31.148 INFO:teuthology.orchestra.run.gibba017.stdout:ID_LIKE="rhel fedora" 2021-03-05T02:51:31.148 INFO:teuthology.orchestra.run.gibba017.stdout:VERSION_ID="8" 2021-03-05T02:51:31.148 INFO:teuthology.orchestra.run.gibba017.stdout:PLATFORM_ID="platform:el8" 2021-03-05T02:51:31.148 INFO:teuthology.orchestra.run.gibba017.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-03-05T02:51:31.149 INFO:teuthology.orchestra.run.gibba017.stdout:ANSI_COLOR="0;31" 2021-03-05T02:51:31.149 INFO:teuthology.orchestra.run.gibba017.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-03-05T02:51:31.149 INFO:teuthology.orchestra.run.gibba017.stdout:HOME_URL="https://www.centos.org/" 2021-03-05T02:51:31.149 INFO:teuthology.orchestra.run.gibba017.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-03-05T02:51:31.150 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-03-05T02:51:31.150 INFO:teuthology.orchestra.run.gibba017.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-03-05T02:51:31.150 INFO:teuthology.orchestra.run.gibba017.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-03-05T02:51:31.150 INFO:teuthology.orchestra.run.gibba017.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-03-05T02:51:31.150 INFO:teuthology.orchestra.run.gibba017.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-03-05T02:51:31.151 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-03-05T02:51:31.152 INFO:teuthology.lock.ops:Updating gibba017.front.sepia.ceph.com on lock server 2021-03-05T02:51:31.167 INFO:teuthology.lock.ops:Updating [gibba039.front.sepia.ceph.com]: set os type and version on server 2021-03-05T02:51:31.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-05T02:51:31.402 INFO:teuthology.orchestra.run.gibba039.stdout:x86_64 2021-03-05T02:51:31.454 INFO:teuthology.orchestra.run.gibba039.stdout:NAME="CentOS Linux" 2021-03-05T02:51:31.454 INFO:teuthology.orchestra.run.gibba039.stdout:VERSION="8 (Core)" 2021-03-05T02:51:31.454 INFO:teuthology.orchestra.run.gibba039.stdout:ID="centos" 2021-03-05T02:51:31.455 INFO:teuthology.orchestra.run.gibba039.stdout:ID_LIKE="rhel fedora" 2021-03-05T02:51:31.455 INFO:teuthology.orchestra.run.gibba039.stdout:VERSION_ID="8" 2021-03-05T02:51:31.455 INFO:teuthology.orchestra.run.gibba039.stdout:PLATFORM_ID="platform:el8" 2021-03-05T02:51:31.455 INFO:teuthology.orchestra.run.gibba039.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-03-05T02:51:31.456 INFO:teuthology.orchestra.run.gibba039.stdout:ANSI_COLOR="0;31" 2021-03-05T02:51:31.456 INFO:teuthology.orchestra.run.gibba039.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-03-05T02:51:31.456 INFO:teuthology.orchestra.run.gibba039.stdout:HOME_URL="https://www.centos.org/" 2021-03-05T02:51:31.456 INFO:teuthology.orchestra.run.gibba039.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-03-05T02:51:31.456 INFO:teuthology.orchestra.run.gibba039.stdout: 2021-03-05T02:51:31.457 INFO:teuthology.orchestra.run.gibba039.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-03-05T02:51:31.457 INFO:teuthology.orchestra.run.gibba039.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-03-05T02:51:31.457 INFO:teuthology.orchestra.run.gibba039.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-03-05T02:51:31.457 INFO:teuthology.orchestra.run.gibba039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-03-05T02:51:31.457 INFO:teuthology.orchestra.run.gibba039.stdout: 2021-03-05T02:51:31.458 INFO:teuthology.lock.ops:Updating gibba039.front.sepia.ceph.com on lock server 2021-03-05T02:51:31.542 INFO:teuthology.dispatcher.supervisor:Running job 5932848 2021-03-05T02:51:31.542 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology -v --owner scheduled_hmunjulu@teuthology --archive /home/teuthworker/archive/hmunjulu-2021-03-04_22:20:10-rbd-master-distro-basic-gibba/5932848 --name hmunjulu-2021-03-04_22:20:10-rbd-master-distro-basic-gibba --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} -- /home/teuthworker/archive/hmunjulu-2021-03-04_22:20:10-rbd-master-distro-basic-gibba/5932848/orig.config.yaml 2021-03-05T02:51:31.547 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/hmunjulu-2021-03-04_22:20:10-rbd-master-distro-basic-gibba/5932848 2021-03-05T02:51:31.548 INFO:teuthology.dispatcher.supervisor:Job PID: 15159 2021-03-05T02:51:31.548 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-03-05T14:52:21.719 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-05T14:52:22.042 INFO:teuthology.kill:Killing Pids: {15159} 2021-03-05T14:52:23.140 INFO:teuthology.kill:Nuking machines: ['gibba017', 'gibba039'] 2021-03-05T14:52:24.101 INFO:teuthology.kill:2021-03-05 14:52:24,100.100 INFO:teuthology.nuke:targets: 2021-03-05T14:52:24.102 INFO:teuthology.kill: gibba017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9kLhDJh5AxwX3L8cKKkNxQu6ygJ6RRhkq7F408TzlokmtjsoOv/NLXiInfr9D8bWz5XdaCQQ7R9UShM/EC6VfKhBTvBfreqgwiURMJRowrYgc0wbl4rd32nrr/0KEYRZJ1omIBFmykE+wP5bnkNFmmJW57T8qmWUzwGxCUgM9B7zMlS6Rr37AVf+7g9SqHilGqHzZaf6hczMqy3XMxmrWoAz24Iiie8pivRc80Czdtb7rol99U6XbpVJhHYPS79VSF7FqysTrQX4vJQiuPPTfpbwLlqCPEDADBaujBNEv5faA26liNGrmf1WopA5NJY4VulITbryDONIApUEdJvi5QdogogXbHxwkoQGWd/i7NoBPie8joJ6aUwA6uhskvLtPGNz0X404t8oAfr1snXscsG1nV292QSWlPFTZfanOItNI4uCOqERN3EIh/XhJoTM2SBekdvrqBkaKALw01Ec6mhmRcVV5dMKF4SeajaV6k1T/SukJNMD26VgW+8rI0fE= 2021-03-05T14:52:24.102 INFO:teuthology.kill: gibba039.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxS894b2us9GzzEbwU5wF91VYb3UtuZgj1aGxqJcfEZpxDfyoY5E64MG14T+3aP8BR4WCH+MCvR48YvLvFchrdkc8MNsDdipw2gNyJ1ktHMQONov2qEUKDAKm7kWeBZs1NVBCFKzhCYC+mQyxx5nH/MGkSTuMn3nLUN5F6Dhcw/PkaD/DjkXPJR6FXMsr+7ESqPLC0tpYciyG4G51v33FATrGeeSfPSV+t+KAKixJyxe0Yg4Olo2TWb7rKN/2o2LD+we/A2bpWkZLglZe1hyX1E8mC8AivHF+Mo8DerHlYSOQnyqU7T+0X5SNsk3ad0n7EeOjjOHwcrFKp5tJ2a5EpCGvSQzOg9kjTrvfG0SeMNiwjroWZ7KiG7ZvIoSzCUgzIELtrkO4+mUohZiTX1HD9uR8WdYoiYhHM26HeDEDWAuOS77mTRhY5fJFzIvAXUh+Jbkottm5ADSFgDShKK4vO5s849H0a4sj2ppugKv4z5lwECqPv/wnZ6fr0cqcdfUE= 2021-03-05T14:52:24.102 INFO:teuthology.kill:2021-03-05 14:52:24,101.101 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-05T14:52:24.105 INFO:teuthology.kill:2021-03-05 14:52:24,103.103 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-05T14:52:24.168 INFO:teuthology.kill:2021-03-05 14:52:24,167.167 INFO:teuthology.orchestra.console:Power off gibba017 2021-03-05T14:52:24.186 INFO:teuthology.kill:2021-03-05 14:52:24,186.186 INFO:teuthology.orchestra.console:Power off gibba039 2021-03-05T14:52:32.553 INFO:teuthology.kill:2021-03-05 14:52:32,553.553 INFO:teuthology.orchestra.console:Power off for gibba017 completed 2021-03-05T14:52:32.558 INFO:teuthology.kill:2021-03-05 14:52:32,558.558 INFO:teuthology.orchestra.console:Power off for gibba039 completed 2021-03-05T14:52:32.827 INFO:teuthology.task.internal:roles: ubuntu@gibba017.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2021-03-05T14:52:32.827 INFO:teuthology.task.internal:roles: ubuntu@gibba039.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2021-03-05T14:52:32.828 INFO:teuthology.misc:Compressing logs... 2021-03-05T14:52:32.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-05T14:53:32.890 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-05T14:53:32.962 INFO:teuthology.kill:No teuthology processes running 2021-03-05T14:53:34.735 INFO:teuthology.kill:Nuking machines: ['gibba017', 'gibba039'] 2021-03-05T14:53:35.709 INFO:teuthology.kill:2021-03-05 14:53:35,708.708 INFO:teuthology.nuke:targets: 2021-03-05T14:53:35.710 INFO:teuthology.kill: gibba017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9kLhDJh5AxwX3L8cKKkNxQu6ygJ6RRhkq7F408TzlokmtjsoOv/NLXiInfr9D8bWz5XdaCQQ7R9UShM/EC6VfKhBTvBfreqgwiURMJRowrYgc0wbl4rd32nrr/0KEYRZJ1omIBFmykE+wP5bnkNFmmJW57T8qmWUzwGxCUgM9B7zMlS6Rr37AVf+7g9SqHilGqHzZaf6hczMqy3XMxmrWoAz24Iiie8pivRc80Czdtb7rol99U6XbpVJhHYPS79VSF7FqysTrQX4vJQiuPPTfpbwLlqCPEDADBaujBNEv5faA26liNGrmf1WopA5NJY4VulITbryDONIApUEdJvi5QdogogXbHxwkoQGWd/i7NoBPie8joJ6aUwA6uhskvLtPGNz0X404t8oAfr1snXscsG1nV292QSWlPFTZfanOItNI4uCOqERN3EIh/XhJoTM2SBekdvrqBkaKALw01Ec6mhmRcVV5dMKF4SeajaV6k1T/SukJNMD26VgW+8rI0fE= 2021-03-05T14:53:35.710 INFO:teuthology.kill: gibba039.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxS894b2us9GzzEbwU5wF91VYb3UtuZgj1aGxqJcfEZpxDfyoY5E64MG14T+3aP8BR4WCH+MCvR48YvLvFchrdkc8MNsDdipw2gNyJ1ktHMQONov2qEUKDAKm7kWeBZs1NVBCFKzhCYC+mQyxx5nH/MGkSTuMn3nLUN5F6Dhcw/PkaD/DjkXPJR6FXMsr+7ESqPLC0tpYciyG4G51v33FATrGeeSfPSV+t+KAKixJyxe0Yg4Olo2TWb7rKN/2o2LD+we/A2bpWkZLglZe1hyX1E8mC8AivHF+Mo8DerHlYSOQnyqU7T+0X5SNsk3ad0n7EeOjjOHwcrFKp5tJ2a5EpCGvSQzOg9kjTrvfG0SeMNiwjroWZ7KiG7ZvIoSzCUgzIELtrkO4+mUohZiTX1HD9uR8WdYoiYhHM26HeDEDWAuOS77mTRhY5fJFzIvAXUh+Jbkottm5ADSFgDShKK4vO5s849H0a4sj2ppugKv4z5lwECqPv/wnZ6fr0cqcdfUE= 2021-03-05T14:53:35.731 INFO:teuthology.kill:2021-03-05 14:53:35,730.730 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-05T14:53:35.733 INFO:teuthology.kill:2021-03-05 14:53:35,733.733 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-05T14:53:35.796 INFO:teuthology.kill:2021-03-05 14:53:35,796.796 INFO:teuthology.orchestra.console:Power off gibba039 2021-03-05T14:53:35.812 INFO:teuthology.kill:2021-03-05 14:53:35,812.812 INFO:teuthology.orchestra.console:Power off gibba017 2021-03-05T14:53:35.974 INFO:teuthology.kill:2021-03-05 14:53:35,974.974 INFO:teuthology.orchestra.console:Power off for gibba017 completed 2021-03-05T14:53:35.981 INFO:teuthology.kill:2021-03-05 14:53:35,980.980 INFO:teuthology.orchestra.console:Power off for gibba039 completed 2021-03-05T14:53:36.116 INFO:teuthology.kill:2021-03-05 14:53:36,115.115 INFO:teuthology.lock.ops:unlocked gibba017.front.sepia.ceph.com 2021-03-05T14:53:36.119 INFO:teuthology.kill:2021-03-05 14:53:36,119.119 INFO:teuthology.lock.ops:unlocked gibba039.front.sepia.ceph.com 2021-03-05T14:55:36.388 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15