2021-06-30T11:38:53.184 INFO:root:teuthology version: 1.1.0-f359b10d 2021-06-30T11:38:53.282 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba029.front.sepia.ceph.com' reimaging 2021-06-30T11:38:53.284 INFO:teuthology.lock.ops:Updating [ubuntu@gibba029.front.sepia.ceph.com]: reset os type and version on server 2021-06-30T11:38:53.284 INFO:teuthology.lock.ops:Updating gibba029.front.sepia.ceph.com on lock server 2021-06-30T11:38:53.299 INFO:teuthology.lock.ops:Node 'ubuntu@gibba029.front.sepia.ceph.com' reimaging is complete 2021-06-30T11:38:53.299 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba043.front.sepia.ceph.com' reimaging 2021-06-30T11:38:53.299 INFO:teuthology.lock.ops:Updating [ubuntu@gibba043.front.sepia.ceph.com]: reset os type and version on server 2021-06-30T11:38:53.300 INFO:teuthology.lock.ops:Updating gibba043.front.sepia.ceph.com on lock server 2021-06-30T11:38:53.316 INFO:teuthology.lock.ops:Node 'ubuntu@gibba043.front.sepia.ceph.com' reimaging is complete 2021-06-30T11:38:53.513 INFO:teuthology.provision.fog.gibba029:Scheduling deploy of rhel 8.4 2021-06-30T11:38:53.524 INFO:teuthology.provision.fog.gibba043:Scheduling deploy of rhel 8.4 2021-06-30T11:38:53.888 INFO:teuthology.orchestra.console:Power off gibba029 2021-06-30T11:38:53.903 INFO:teuthology.orchestra.console:Power off gibba043 2021-06-30T11:39:02.354 INFO:teuthology.orchestra.console:Power off for gibba029 completed 2021-06-30T11:39:02.361 INFO:teuthology.orchestra.console:Power off for gibba043 completed 2021-06-30T11:39:02.456 INFO:teuthology.orchestra.console:Power on gibba029 2021-06-30T11:39:02.465 INFO:teuthology.orchestra.console:Power on gibba043 2021-06-30T11:39:10.844 INFO:teuthology.orchestra.console:Power on for gibba029 completed 2021-06-30T11:39:10.845 INFO:teuthology.orchestra.console:Power on for gibba043 completed 2021-06-30T11:39:10.945 INFO:teuthology.provision.fog.gibba029:Waiting for deploy to finish 2021-06-30T11:39:10.947 INFO:teuthology.provision.fog.gibba043:Waiting for deploy to finish 2021-06-30T11:43:11.772 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-30T11:43:26.896 INFO:teuthology.provision.fog.gibba029:Node is ready 2021-06-30T11:43:26.909 INFO:teuthology.orchestra.run.gibba029.stdout:gibba029.front.sepia.ceph.com 2021-06-30T11:43:26.962 INFO:teuthology.orchestra.run.gibba029.stdout:172.21.2.129 gibba029.front.sepia.ceph.com gibba029 2021-06-30T11:43:27.270 INFO:teuthology.provision.fog.gibba029:Deploy complete! 2021-06-30T11:43:46.521 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-30T11:43:46.695 INFO:teuthology.provision.fog.gibba043:Node is ready 2021-06-30T11:43:46.747 INFO:teuthology.orchestra.run.gibba043.stdout:gibba043.front.sepia.ceph.com 2021-06-30T11:43:46.799 INFO:teuthology.orchestra.run.gibba043.stdout:172.21.2.143 gibba043.front.sepia.ceph.com gibba043 2021-06-30T11:43:47.088 INFO:teuthology.provision.fog.gibba043:Deploy complete! 2021-06-30T11:43:47.219 INFO:teuthology.lock.ops:Checking gibba029.front.sepia.ceph.com 2021-06-30T11:43:47.220 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-30T11:43:47.359 INFO:teuthology.lock.ops:Checking gibba043.front.sepia.ceph.com 2021-06-30T11:43:47.359 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-30T11:43:47.374 INFO:teuthology.lock.ops:Updating [gibba029.front.sepia.ceph.com]: set os type and version on server 2021-06-30T11:43:47.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-30T11:43:47.578 INFO:teuthology.orchestra.run.gibba029.stdout:x86_64 2021-06-30T11:43:47.631 INFO:teuthology.orchestra.run.gibba029.stdout:NAME="Red Hat Enterprise Linux" 2021-06-30T11:43:47.631 INFO:teuthology.orchestra.run.gibba029.stdout:VERSION="8.4 (Ootpa)" 2021-06-30T11:43:47.632 INFO:teuthology.orchestra.run.gibba029.stdout:ID="rhel" 2021-06-30T11:43:47.632 INFO:teuthology.orchestra.run.gibba029.stdout:ID_LIKE="fedora" 2021-06-30T11:43:47.632 INFO:teuthology.orchestra.run.gibba029.stdout:VERSION_ID="8.4" 2021-06-30T11:43:47.632 INFO:teuthology.orchestra.run.gibba029.stdout:PLATFORM_ID="platform:el8" 2021-06-30T11:43:47.633 INFO:teuthology.orchestra.run.gibba029.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2021-06-30T11:43:47.633 INFO:teuthology.orchestra.run.gibba029.stdout:ANSI_COLOR="0;31" 2021-06-30T11:43:47.633 INFO:teuthology.orchestra.run.gibba029.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2021-06-30T11:43:47.633 INFO:teuthology.orchestra.run.gibba029.stdout:HOME_URL="https://www.redhat.com/" 2021-06-30T11:43:47.634 INFO:teuthology.orchestra.run.gibba029.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2021-06-30T11:43:47.634 INFO:teuthology.orchestra.run.gibba029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-06-30T11:43:47.634 INFO:teuthology.orchestra.run.gibba029.stdout: 2021-06-30T11:43:47.634 INFO:teuthology.orchestra.run.gibba029.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-06-30T11:43:47.635 INFO:teuthology.orchestra.run.gibba029.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2021-06-30T11:43:47.635 INFO:teuthology.orchestra.run.gibba029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-06-30T11:43:47.635 INFO:teuthology.orchestra.run.gibba029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2021-06-30T11:43:47.637 INFO:teuthology.lock.ops:Updating gibba029.front.sepia.ceph.com on lock server 2021-06-30T11:43:47.653 INFO:teuthology.lock.ops:Updating [gibba043.front.sepia.ceph.com]: set os type and version on server 2021-06-30T11:43:47.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-30T11:43:47.854 INFO:teuthology.orchestra.run.gibba043.stdout:x86_64 2021-06-30T11:43:47.906 INFO:teuthology.orchestra.run.gibba043.stdout:NAME="Red Hat Enterprise Linux" 2021-06-30T11:43:47.907 INFO:teuthology.orchestra.run.gibba043.stdout:VERSION="8.4 (Ootpa)" 2021-06-30T11:43:47.907 INFO:teuthology.orchestra.run.gibba043.stdout:ID="rhel" 2021-06-30T11:43:47.908 INFO:teuthology.orchestra.run.gibba043.stdout:ID_LIKE="fedora" 2021-06-30T11:43:47.908 INFO:teuthology.orchestra.run.gibba043.stdout:VERSION_ID="8.4" 2021-06-30T11:43:47.909 INFO:teuthology.orchestra.run.gibba043.stdout:PLATFORM_ID="platform:el8" 2021-06-30T11:43:47.909 INFO:teuthology.orchestra.run.gibba043.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2021-06-30T11:43:47.910 INFO:teuthology.orchestra.run.gibba043.stdout:ANSI_COLOR="0;31" 2021-06-30T11:43:47.910 INFO:teuthology.orchestra.run.gibba043.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2021-06-30T11:43:47.910 INFO:teuthology.orchestra.run.gibba043.stdout:HOME_URL="https://www.redhat.com/" 2021-06-30T11:43:47.911 INFO:teuthology.orchestra.run.gibba043.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2021-06-30T11:43:47.911 INFO:teuthology.orchestra.run.gibba043.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-06-30T11:43:47.912 INFO:teuthology.orchestra.run.gibba043.stdout: 2021-06-30T11:43:47.912 INFO:teuthology.orchestra.run.gibba043.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-06-30T11:43:47.913 INFO:teuthology.orchestra.run.gibba043.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2021-06-30T11:43:47.913 INFO:teuthology.orchestra.run.gibba043.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-06-30T11:43:47.914 INFO:teuthology.orchestra.run.gibba043.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2021-06-30T11:43:47.915 INFO:teuthology.lock.ops:Updating gibba043.front.sepia.ceph.com on lock server 2021-06-30T11:43:48.018 INFO:teuthology.dispatcher.supervisor:Running job 6182146 2021-06-30T11:43:48.018 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-06-21_02:01:03-rbd-master-distro-basic-gibba/6182146 --name teuthology-2021-06-21_02:01:03-rbd-master-distro-basic-gibba --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} -- /home/teuthworker/archive/teuthology-2021-06-21_02:01:03-rbd-master-distro-basic-gibba/6182146/orig.config.yaml 2021-06-30T11:43:48.026 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-06-21_02:01:03-rbd-master-distro-basic-gibba/6182146 2021-06-30T11:43:48.027 INFO:teuthology.dispatcher.supervisor:Job PID: 23622 2021-06-30T11:43:48.028 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-06-30T23:44:41.259 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-06-30T23:44:41.544 INFO:teuthology.kill:Killing Pids: {23622} 2021-06-30T23:44:43.210 INFO:teuthology.kill:Nuking machines: ['gibba029', 'gibba043'] 2021-06-30T23:44:44.392 INFO:teuthology.kill:2021-06-30 23:44:44,392.392 INFO:teuthology.nuke:targets: 2021-06-30T23:44:44.393 INFO:teuthology.kill: gibba029.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGPkoCyEvsZtjkA1aHEA0mfXA7v3RXHbfjW0b5tDiw0DLG2uWqQOpmhiqFTzkawzB9tQn9kNE/c1YZqy2wP9GPkyG7FPVrxNx5MYYL1HeK/TJ1WnP56IlpCIXBf+6Fzv5ylcGkMp8wShrM3o6nKrl9iBrtHUNedn5BC+p3VvUmazVjLz5CnnOqraq3IUVp4SSpzPadHwWW9cf/fp0exbke4QzNhZlJ5c07vM+vkndtLooPSwmGNpFKRP3KICUSHDyZWThOD3byRsvinbuqfJrqT8qlEtf+/K19mwZjElZAnrR8Zj+LeMe1GkdrnzyGaruihADdISgMKnRNGbCnYXYXwREHqC5C7wEVR6Dv+hI9WXsD8OivutnRW36hp29ncDIMxIi4M7g1O+X+ZToY0XyP/zkY/h/z1icpJoi1ptFMfxHtlvAw6De2qCH7xVQhzCGZMqSAqZeeyPmkLyodBuNAVSF260CewO/J05fED5dhtusB8dcigYVcB+A6Tb+wFck= 2021-06-30T23:44:44.393 INFO:teuthology.kill: gibba043.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0UXvPlqtJHwINk3Tj0vjZA3JFgc1nSopfFEVAOEf9W1G+qPjoH56aN0XCUICeHyqdpuEGlsWI3HvfnBHRa/1i1L6cmJosuHKPRQQy9pr6RWyh9xH3AkewEBqLaBYTnBnD/klnKyD2qYFNNKgFL+e+cvrYa88Y9OpQ4PQD55jLMcTp/QfFL54kOPaEKgTLttP81/hk46+3OT3xlyZkvIFfTvFTepr3Rv71Qp2C//o4hDN7UQFzg9pwpS9uBPe1090ZO3GFWBSM283T7OGPDKlRSS4bx0o5YfEhxg2P22D6fD9VpWiseyB7c92xDHLY1uizEEhXDbv3QYpPuGCxoSBcO6XzH9NWdYA4gFbcMKy/LX2QzMtt3+6llxTsjzPYiSQEn1gVwVJURT1QcIwxdgPJWo8WBw/d/n0g4LjVr6P4sFLjtTep6EOSB5cTA9HPWqNykF/CqT8YLv8B43YIrd4bwTnmJuD3XdJeU7YwznEoWa2GQWoi346SJaMiK071MyU= 2021-06-30T23:44:44.394 INFO:teuthology.kill:2021-06-30 23:44:44,392.392 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-30T23:44:44.395 INFO:teuthology.kill:2021-06-30 23:44:44,395.395 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-30T23:44:44.470 INFO:teuthology.kill:2021-06-30 23:44:44,470.470 INFO:teuthology.orchestra.console:Power off gibba043 2021-06-30T23:44:44.491 INFO:teuthology.kill:2021-06-30 23:44:44,491.491 INFO:teuthology.orchestra.console:Power off gibba029 2021-06-30T23:44:52.883 INFO:teuthology.kill:2021-06-30 23:44:52,883.883 INFO:teuthology.orchestra.console:Power off for gibba029 completed 2021-06-30T23:44:56.975 INFO:teuthology.kill:2021-06-30 23:44:56,974.974 INFO:teuthology.orchestra.console:Power off for gibba043 completed 2021-06-30T23:44:57.276 INFO:teuthology.task.internal:roles: ubuntu@gibba029.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2021-06-30T23:44:57.277 INFO:teuthology.task.internal:roles: ubuntu@gibba043.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2021-06-30T23:44:57.277 INFO:teuthology.misc:Compressing logs... 2021-06-30T23:44:57.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-30T23:45:57.347 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-06-30T23:45:57.484 INFO:teuthology.kill:No teuthology processes running 2021-06-30T23:45:59.350 INFO:teuthology.kill:Nuking machines: ['gibba029', 'gibba043'] 2021-06-30T23:46:00.758 INFO:teuthology.kill:2021-06-30 23:46:00,757.757 INFO:teuthology.nuke:targets: 2021-06-30T23:46:00.759 INFO:teuthology.kill: gibba029.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGPkoCyEvsZtjkA1aHEA0mfXA7v3RXHbfjW0b5tDiw0DLG2uWqQOpmhiqFTzkawzB9tQn9kNE/c1YZqy2wP9GPkyG7FPVrxNx5MYYL1HeK/TJ1WnP56IlpCIXBf+6Fzv5ylcGkMp8wShrM3o6nKrl9iBrtHUNedn5BC+p3VvUmazVjLz5CnnOqraq3IUVp4SSpzPadHwWW9cf/fp0exbke4QzNhZlJ5c07vM+vkndtLooPSwmGNpFKRP3KICUSHDyZWThOD3byRsvinbuqfJrqT8qlEtf+/K19mwZjElZAnrR8Zj+LeMe1GkdrnzyGaruihADdISgMKnRNGbCnYXYXwREHqC5C7wEVR6Dv+hI9WXsD8OivutnRW36hp29ncDIMxIi4M7g1O+X+ZToY0XyP/zkY/h/z1icpJoi1ptFMfxHtlvAw6De2qCH7xVQhzCGZMqSAqZeeyPmkLyodBuNAVSF260CewO/J05fED5dhtusB8dcigYVcB+A6Tb+wFck= 2021-06-30T23:46:00.759 INFO:teuthology.kill: gibba043.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0UXvPlqtJHwINk3Tj0vjZA3JFgc1nSopfFEVAOEf9W1G+qPjoH56aN0XCUICeHyqdpuEGlsWI3HvfnBHRa/1i1L6cmJosuHKPRQQy9pr6RWyh9xH3AkewEBqLaBYTnBnD/klnKyD2qYFNNKgFL+e+cvrYa88Y9OpQ4PQD55jLMcTp/QfFL54kOPaEKgTLttP81/hk46+3OT3xlyZkvIFfTvFTepr3Rv71Qp2C//o4hDN7UQFzg9pwpS9uBPe1090ZO3GFWBSM283T7OGPDKlRSS4bx0o5YfEhxg2P22D6fD9VpWiseyB7c92xDHLY1uizEEhXDbv3QYpPuGCxoSBcO6XzH9NWdYA4gFbcMKy/LX2QzMtt3+6llxTsjzPYiSQEn1gVwVJURT1QcIwxdgPJWo8WBw/d/n0g4LjVr6P4sFLjtTep6EOSB5cTA9HPWqNykF/CqT8YLv8B43YIrd4bwTnmJuD3XdJeU7YwznEoWa2GQWoi346SJaMiK071MyU= 2021-06-30T23:46:00.786 INFO:teuthology.kill:2021-06-30 23:46:00,786.786 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-30T23:46:00.790 INFO:teuthology.kill:2021-06-30 23:46:00,790.790 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-30T23:46:00.851 INFO:teuthology.kill:2021-06-30 23:46:00,851.851 INFO:teuthology.orchestra.console:Power off gibba029 2021-06-30T23:46:00.872 INFO:teuthology.kill:2021-06-30 23:46:00,872.872 INFO:teuthology.orchestra.console:Power off gibba043 2021-06-30T23:46:01.026 INFO:teuthology.kill:2021-06-30 23:46:01,025.025 INFO:teuthology.orchestra.console:Power off for gibba029 completed 2021-06-30T23:46:01.042 INFO:teuthology.kill:2021-06-30 23:46:01,042.042 INFO:teuthology.orchestra.console:Power off for gibba043 completed 2021-06-30T23:46:01.178 INFO:teuthology.kill:2021-06-30 23:46:01,177.177 INFO:teuthology.lock.ops:unlocked gibba029.front.sepia.ceph.com 2021-06-30T23:46:01.197 INFO:teuthology.kill:2021-06-30 23:46:01,196.196 INFO:teuthology.lock.ops:unlocked gibba043.front.sepia.ceph.com 2021-06-30T23:48:01.689 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-06-30T23:48:01.847 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-06-30T23:48:01.848 INFO:teuthology.nuke:Checking targets against current locks 2021-06-30T23:48:01.933 INFO:teuthology.nuke:Not nuking gibba043.front.sepia.ceph.com because description doesn't match 2021-06-30T23:48:01.934 INFO:teuthology.nuke:Not nuking gibba029.front.sepia.ceph.com because description doesn't match