2021-04-02T01:03:57.580 INFO:root:teuthology version: 1.1.0-85d61eae 2021-04-02T01:03:57.703 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba011.front.sepia.ceph.com' reimaging 2021-04-02T01:03:57.705 INFO:teuthology.lock.ops:Updating [ubuntu@gibba011.front.sepia.ceph.com]: reset os type and version on server 2021-04-02T01:03:57.706 INFO:teuthology.lock.ops:Updating gibba011.front.sepia.ceph.com on lock server 2021-04-02T01:03:57.724 INFO:teuthology.lock.ops:Node 'ubuntu@gibba011.front.sepia.ceph.com' reimaging is complete 2021-04-02T01:03:57.724 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba017.front.sepia.ceph.com' reimaging 2021-04-02T01:03:57.724 INFO:teuthology.lock.ops:Updating [ubuntu@gibba017.front.sepia.ceph.com]: reset os type and version on server 2021-04-02T01:03:57.725 INFO:teuthology.lock.ops:Updating gibba017.front.sepia.ceph.com on lock server 2021-04-02T01:03:57.742 INFO:teuthology.lock.ops:Node 'ubuntu@gibba017.front.sepia.ceph.com' reimaging is complete 2021-04-02T01:03:58.045 INFO:teuthology.provision.fog.gibba011:Scheduling deploy of centos 8.1 2021-04-02T01:03:58.079 INFO:teuthology.provision.fog.gibba017:Scheduling deploy of centos 8.1 2021-04-02T01:03:58.561 INFO:teuthology.orchestra.console:Power off gibba011 2021-04-02T01:03:58.686 INFO:teuthology.orchestra.console:Power off gibba017 2021-04-02T01:04:06.986 INFO:teuthology.orchestra.console:Power off for gibba011 completed 2021-04-02T01:04:07.087 INFO:teuthology.orchestra.console:Power on gibba011 2021-04-02T01:04:11.196 INFO:teuthology.orchestra.console:Power off for gibba017 completed 2021-04-02T01:04:11.298 INFO:teuthology.orchestra.console:Power on gibba017 2021-04-02T01:04:15.473 INFO:teuthology.orchestra.console:Power on for gibba011 completed 2021-04-02T01:04:15.574 INFO:teuthology.provision.fog.gibba011:Waiting for deploy to finish 2021-04-02T01:04:19.677 INFO:teuthology.orchestra.console:Power on for gibba017 completed 2021-04-02T01:04:19.778 INFO:teuthology.provision.fog.gibba017:Waiting for deploy to finish 2021-04-02T01:08:16.848 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-02T01:08:27.020 INFO:teuthology.provision.fog.gibba011:Node is ready 2021-04-02T01:08:27.036 INFO:teuthology.orchestra.run.gibba011.stdout:gibba011.front.sepia.ceph.com 2021-04-02T01:08:27.088 INFO:teuthology.orchestra.run.gibba011.stdout:172.21.2.111 gibba011.front.sepia.ceph.com gibba011 2021-04-02T01:08:27.410 INFO:teuthology.provision.fog.gibba011:Deploy complete! 2021-04-02T01:08:46.274 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-02T01:08:46.526 INFO:teuthology.provision.fog.gibba017:Node is ready 2021-04-02T01:08:46.579 INFO:teuthology.orchestra.run.gibba017.stdout:gibba017.front.sepia.ceph.com 2021-04-02T01:08:46.633 INFO:teuthology.orchestra.run.gibba017.stdout:172.21.2.117 gibba017.front.sepia.ceph.com gibba017 2021-04-02T01:08:46.956 INFO:teuthology.provision.fog.gibba017:Deploy complete! 2021-04-02T01:08:47.083 INFO:teuthology.lock.ops:Checking gibba011.front.sepia.ceph.com 2021-04-02T01:08:47.084 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-02T01:08:47.102 INFO:teuthology.lock.ops:Checking gibba017.front.sepia.ceph.com 2021-04-02T01:08:47.102 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-02T01:08:47.117 INFO:teuthology.lock.ops:Updating [gibba011.front.sepia.ceph.com]: set os type and version on server 2021-04-02T01:08:47.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-02T01:08:47.368 INFO:teuthology.orchestra.run.gibba011.stdout:x86_64 2021-04-02T01:08:47.421 INFO:teuthology.orchestra.run.gibba011.stdout:NAME="CentOS Linux" 2021-04-02T01:08:47.421 INFO:teuthology.orchestra.run.gibba011.stdout:VERSION="8 (Core)" 2021-04-02T01:08:47.421 INFO:teuthology.orchestra.run.gibba011.stdout:ID="centos" 2021-04-02T01:08:47.421 INFO:teuthology.orchestra.run.gibba011.stdout:ID_LIKE="rhel fedora" 2021-04-02T01:08:47.422 INFO:teuthology.orchestra.run.gibba011.stdout:VERSION_ID="8" 2021-04-02T01:08:47.422 INFO:teuthology.orchestra.run.gibba011.stdout:PLATFORM_ID="platform:el8" 2021-04-02T01:08:47.422 INFO:teuthology.orchestra.run.gibba011.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-04-02T01:08:47.423 INFO:teuthology.orchestra.run.gibba011.stdout:ANSI_COLOR="0;31" 2021-04-02T01:08:47.423 INFO:teuthology.orchestra.run.gibba011.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-04-02T01:08:47.424 INFO:teuthology.orchestra.run.gibba011.stdout:HOME_URL="https://www.centos.org/" 2021-04-02T01:08:47.424 INFO:teuthology.orchestra.run.gibba011.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-04-02T01:08:47.424 INFO:teuthology.orchestra.run.gibba011.stdout: 2021-04-02T01:08:47.424 INFO:teuthology.orchestra.run.gibba011.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-04-02T01:08:47.425 INFO:teuthology.orchestra.run.gibba011.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-04-02T01:08:47.425 INFO:teuthology.orchestra.run.gibba011.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-04-02T01:08:47.425 INFO:teuthology.orchestra.run.gibba011.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-04-02T01:08:47.425 INFO:teuthology.orchestra.run.gibba011.stdout: 2021-04-02T01:08:47.427 INFO:teuthology.lock.ops:Updating gibba011.front.sepia.ceph.com on lock server 2021-04-02T01:08:47.444 INFO:teuthology.lock.ops:Updating [gibba017.front.sepia.ceph.com]: set os type and version on server 2021-04-02T01:08:47.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-02T01:08:47.695 INFO:teuthology.orchestra.run.gibba017.stdout:x86_64 2021-04-02T01:08:47.749 INFO:teuthology.orchestra.run.gibba017.stdout:NAME="CentOS Linux" 2021-04-02T01:08:47.749 INFO:teuthology.orchestra.run.gibba017.stdout:VERSION="8 (Core)" 2021-04-02T01:08:47.749 INFO:teuthology.orchestra.run.gibba017.stdout:ID="centos" 2021-04-02T01:08:47.749 INFO:teuthology.orchestra.run.gibba017.stdout:ID_LIKE="rhel fedora" 2021-04-02T01:08:47.750 INFO:teuthology.orchestra.run.gibba017.stdout:VERSION_ID="8" 2021-04-02T01:08:47.750 INFO:teuthology.orchestra.run.gibba017.stdout:PLATFORM_ID="platform:el8" 2021-04-02T01:08:47.750 INFO:teuthology.orchestra.run.gibba017.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-04-02T01:08:47.750 INFO:teuthology.orchestra.run.gibba017.stdout:ANSI_COLOR="0;31" 2021-04-02T01:08:47.751 INFO:teuthology.orchestra.run.gibba017.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-04-02T01:08:47.751 INFO:teuthology.orchestra.run.gibba017.stdout:HOME_URL="https://www.centos.org/" 2021-04-02T01:08:47.751 INFO:teuthology.orchestra.run.gibba017.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-04-02T01:08:47.751 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-04-02T01:08:47.752 INFO:teuthology.orchestra.run.gibba017.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-04-02T01:08:47.753 INFO:teuthology.orchestra.run.gibba017.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-04-02T01:08:47.753 INFO:teuthology.orchestra.run.gibba017.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-04-02T01:08:47.753 INFO:teuthology.orchestra.run.gibba017.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-04-02T01:08:47.753 INFO:teuthology.orchestra.run.gibba017.stdout: 2021-04-02T01:08:47.754 INFO:teuthology.lock.ops:Updating gibba017.front.sepia.ceph.com on lock server 2021-04-02T01:08:47.838 INFO:teuthology.dispatcher.supervisor:Running job 6014561 2021-04-02T01:08:47.839 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-04-01_06:00:03-rbd-octopus-distro-basic-gibba/6014561 --name teuthology-2021-04-01_06:00:03-rbd-octopus-distro-basic-gibba --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} -- /home/teuthworker/archive/teuthology-2021-04-01_06:00:03-rbd-octopus-distro-basic-gibba/6014561/orig.config.yaml 2021-04-02T01:08:47.842 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-04-01_06:00:03-rbd-octopus-distro-basic-gibba/6014561 2021-04-02T01:08:47.843 INFO:teuthology.dispatcher.supervisor:Job PID: 17139 2021-04-02T01:08:47.843 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-04-02T13:09:55.442 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-04-02T13:09:55.667 INFO:teuthology.kill:Killing Pids: {17139} 2021-04-02T13:09:56.641 INFO:teuthology.kill:Nuking machines: ['gibba011', 'gibba017'] 2021-04-02T13:09:57.466 INFO:teuthology.kill:2021-04-02 13:09:57,466.466 INFO:teuthology.nuke:targets: 2021-04-02T13:09:57.467 INFO:teuthology.kill: gibba011.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/sawcD4gNMbd/6i05eonnBWObkfjM7XQiPRw790GEjDi+fRwxkm64SiJvv8sFfiMNl+QWPvU//6qlTbPnCWmVgAMW4eaITTjwiqCy8043g0h/tzCaiRVARGeeS/LR3Olu80LJrXwfPtAcxEl3SUR/ub8JcEEz2x91Zz+STOMt6Qo6J+W4RUCGwhZPaj4VqZqenpbI7rACNVpTCEZKH6CzOVRW99fEkXvQTqFXCQnzLyj6tperFcAS5SZRhEAxaLkgC3qP+EXrXqEYxaZW5wUvlwuBOgFe4Z2Rjqv7b8Fb3ELb9K36jJi9yNKFKw2p0bUky5TVXXlGH5+Js3U5ZG9TDLyjYS27cXp6gcexJzCt5lUCxrwnDxqlix3QQxAx0obNGSeDtsDcB+wyYP1Sk8XthrwPUotIFA9tj/pjiR/6QKZDT1qFqZrayhCbcW7WIU7urq3aFi0SpnL679hTsjkoz9RQLZlVzbi9mawy2sUD1pvdycoitkbTeQFu+OEdnSM= 2021-04-02T13:09:57.468 INFO:teuthology.kill: gibba017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFihBkiSylpN/+5L9jEjuBnpdFAJtvxaWs4VmtheuClgE+s0SM+C2dhXhqaBITLclHW6qzljw2aB/a6CPEK1iPurkgOcCD8kgbuk8sduNMaDoGCqWf+Xnt2ouB/VFq/3ATmbEWbiCb8ZzBPRjuh+NnT5qOEro5ncPn794SKzgUaDd3uN/Zf+Hs6fzW/XFl5C8q1RiGWmVPLUJWQalmMAlBDz1rrMyleASFNCKZBU1DrLGM3a95L6MfMFXtmnChJKp5MZ7dli5GOCajtLS0RaC8QuXFuT6V0tz01QvmzMy3QjbvzgSh0h5RGYUPqEsbm72P/QCtYJR4bmB0AI2XH8vof9N52ZSYjwkN+R0jzzr6wjraqvLr5d2NoS8skbMg/0r9FY3ynk6Cb5dOUfBg1d43UEhk7hylogTv+WE89w3i+sarvZPCiDegzHdUIFNh9et8mu1udGpEbfkncgPGOwtzOLkkqlhjRgjIUS6T2U496kzaSAMq88K6KjLhfsHoRCE= 2021-04-02T13:09:57.468 INFO:teuthology.kill:2021-04-02 13:09:57,466.466 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-02T13:09:57.469 INFO:teuthology.kill:2021-04-02 13:09:57,468.468 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-02T13:09:57.540 INFO:teuthology.kill:2021-04-02 13:09:57,540.540 INFO:teuthology.orchestra.console:Power off gibba011 2021-04-02T13:09:57.552 INFO:teuthology.kill:2021-04-02 13:09:57,552.552 INFO:teuthology.orchestra.console:Power off gibba017 2021-04-02T13:10:05.919 INFO:teuthology.kill:2021-04-02 13:10:05,918.918 INFO:teuthology.orchestra.console:Power off for gibba011 completed 2021-04-02T13:10:10.034 INFO:teuthology.kill:2021-04-02 13:10:10,033.033 INFO:teuthology.orchestra.console:Power off for gibba017 completed 2021-04-02T13:10:10.230 INFO:teuthology.task.internal:roles: ubuntu@gibba011.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2021-04-02T13:10:10.230 INFO:teuthology.task.internal:roles: ubuntu@gibba017.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2021-04-02T13:10:10.231 INFO:teuthology.misc:Compressing logs... 2021-04-02T13:10:10.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-02T13:10:44.827 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-04-02T13:10:44.887 INFO:teuthology.kill:No teuthology processes running 2021-04-02T13:10:45.811 INFO:teuthology.kill:Nuking machines: ['gibba011', 'gibba017'] 2021-04-02T13:10:46.642 INFO:teuthology.kill:2021-04-02 13:10:46,641.641 INFO:teuthology.nuke:targets: 2021-04-02T13:10:46.643 INFO:teuthology.kill: gibba011.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/sawcD4gNMbd/6i05eonnBWObkfjM7XQiPRw790GEjDi+fRwxkm64SiJvv8sFfiMNl+QWPvU//6qlTbPnCWmVgAMW4eaITTjwiqCy8043g0h/tzCaiRVARGeeS/LR3Olu80LJrXwfPtAcxEl3SUR/ub8JcEEz2x91Zz+STOMt6Qo6J+W4RUCGwhZPaj4VqZqenpbI7rACNVpTCEZKH6CzOVRW99fEkXvQTqFXCQnzLyj6tperFcAS5SZRhEAxaLkgC3qP+EXrXqEYxaZW5wUvlwuBOgFe4Z2Rjqv7b8Fb3ELb9K36jJi9yNKFKw2p0bUky5TVXXlGH5+Js3U5ZG9TDLyjYS27cXp6gcexJzCt5lUCxrwnDxqlix3QQxAx0obNGSeDtsDcB+wyYP1Sk8XthrwPUotIFA9tj/pjiR/6QKZDT1qFqZrayhCbcW7WIU7urq3aFi0SpnL679hTsjkoz9RQLZlVzbi9mawy2sUD1pvdycoitkbTeQFu+OEdnSM= 2021-04-02T13:10:46.643 INFO:teuthology.kill: gibba017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFihBkiSylpN/+5L9jEjuBnpdFAJtvxaWs4VmtheuClgE+s0SM+C2dhXhqaBITLclHW6qzljw2aB/a6CPEK1iPurkgOcCD8kgbuk8sduNMaDoGCqWf+Xnt2ouB/VFq/3ATmbEWbiCb8ZzBPRjuh+NnT5qOEro5ncPn794SKzgUaDd3uN/Zf+Hs6fzW/XFl5C8q1RiGWmVPLUJWQalmMAlBDz1rrMyleASFNCKZBU1DrLGM3a95L6MfMFXtmnChJKp5MZ7dli5GOCajtLS0RaC8QuXFuT6V0tz01QvmzMy3QjbvzgSh0h5RGYUPqEsbm72P/QCtYJR4bmB0AI2XH8vof9N52ZSYjwkN+R0jzzr6wjraqvLr5d2NoS8skbMg/0r9FY3ynk6Cb5dOUfBg1d43UEhk7hylogTv+WE89w3i+sarvZPCiDegzHdUIFNh9et8mu1udGpEbfkncgPGOwtzOLkkqlhjRgjIUS6T2U496kzaSAMq88K6KjLhfsHoRCE= 2021-04-02T13:10:46.663 INFO:teuthology.kill:2021-04-02 13:10:46,663.663 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-02T13:10:46.676 INFO:teuthology.kill:2021-04-02 13:10:46,675.675 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-02T13:10:46.741 INFO:teuthology.kill:2021-04-02 13:10:46,741.741 INFO:teuthology.orchestra.console:Power off gibba011 2021-04-02T13:10:46.754 INFO:teuthology.kill:2021-04-02 13:10:46,754.754 INFO:teuthology.orchestra.console:Power off gibba017 2021-04-02T13:10:46.885 INFO:teuthology.kill:2021-04-02 13:10:46,885.885 INFO:teuthology.orchestra.console:Power off for gibba011 completed 2021-04-02T13:10:46.900 INFO:teuthology.kill:2021-04-02 13:10:46,900.900 INFO:teuthology.orchestra.console:Power off for gibba017 completed 2021-04-02T13:10:47.043 INFO:teuthology.kill:2021-04-02 13:10:47,043.043 INFO:teuthology.lock.ops:unlocked gibba011.front.sepia.ceph.com 2021-04-02T13:10:47.060 INFO:teuthology.kill:2021-04-02 13:10:47,060.060 INFO:teuthology.lock.ops:unlocked gibba017.front.sepia.ceph.com 2021-04-02T13:12:47.317 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-04-02T13:12:47.407 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-04-02T13:12:47.407 INFO:teuthology.nuke:Checking targets against current locks 2021-04-02T13:12:47.496 INFO:teuthology.nuke:Not nuking gibba017.front.sepia.ceph.com because description doesn't match 2021-04-02T13:12:47.496 INFO:teuthology.nuke:Not nuking gibba011.front.sepia.ceph.com because description doesn't match