2021-05-21T15:04:57.490 INFO:root:teuthology version: 1.1.0-471a2f1e 2021-05-21T15:04:57.727 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba041.front.sepia.ceph.com' reimaging 2021-05-21T15:04:57.729 INFO:teuthology.lock.ops:Updating [ubuntu@gibba041.front.sepia.ceph.com]: reset os type and version on server 2021-05-21T15:04:57.729 INFO:teuthology.lock.ops:Updating gibba041.front.sepia.ceph.com on lock server 2021-05-21T15:04:57.747 INFO:teuthology.lock.ops:Node 'ubuntu@gibba041.front.sepia.ceph.com' reimaging is complete 2021-05-21T15:04:57.748 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba042.front.sepia.ceph.com' reimaging 2021-05-21T15:04:57.748 INFO:teuthology.lock.ops:Updating [ubuntu@gibba042.front.sepia.ceph.com]: reset os type and version on server 2021-05-21T15:04:57.749 INFO:teuthology.lock.ops:Updating gibba042.front.sepia.ceph.com on lock server 2021-05-21T15:04:57.781 INFO:teuthology.lock.ops:Node 'ubuntu@gibba042.front.sepia.ceph.com' reimaging is complete 2021-05-21T15:04:57.988 INFO:teuthology.provision.fog.gibba041:Scheduling deploy of centos 8.3 2021-05-21T15:04:58.015 INFO:teuthology.provision.fog.gibba042:Scheduling deploy of centos 8.3 2021-05-21T15:04:58.392 INFO:teuthology.orchestra.console:Power off gibba041 2021-05-21T15:04:58.559 INFO:teuthology.orchestra.console:Power off gibba042 2021-05-21T15:05:07.058 INFO:teuthology.orchestra.console:Power off for gibba041 completed 2021-05-21T15:05:07.079 INFO:teuthology.orchestra.console:Power off for gibba042 completed 2021-05-21T15:05:07.160 INFO:teuthology.orchestra.console:Power on gibba041 2021-05-21T15:05:07.195 INFO:teuthology.orchestra.console:Power on gibba042 2021-05-21T15:05:15.826 INFO:teuthology.orchestra.console:Power on for gibba041 completed 2021-05-21T15:05:15.831 INFO:teuthology.orchestra.console:Power on for gibba042 completed 2021-05-21T15:05:15.932 INFO:teuthology.provision.fog.gibba041:Waiting for deploy to finish 2021-05-21T15:05:15.936 INFO:teuthology.provision.fog.gibba042:Waiting for deploy to finish 2021-05-21T15:09:42.118 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-21T15:09:42.152 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-21T15:09:42.400 INFO:teuthology.provision.fog.gibba042:Node is ready 2021-05-21T15:09:42.411 INFO:teuthology.orchestra.run.gibba042.stdout:gibba042.front.sepia.ceph.com 2021-05-21T15:09:42.463 INFO:teuthology.provision.fog.gibba041:Node is ready 2021-05-21T15:09:42.464 INFO:teuthology.orchestra.run.gibba042.stdout:172.21.2.142 gibba042.front.sepia.ceph.com gibba042 2021-05-21T15:09:42.514 INFO:teuthology.orchestra.run.gibba041.stdout:gibba041.front.sepia.ceph.com 2021-05-21T15:09:42.567 INFO:teuthology.orchestra.run.gibba041.stdout:172.21.2.141 gibba041.front.sepia.ceph.com gibba041 2021-05-21T15:09:42.747 INFO:teuthology.provision.fog.gibba042:Deploy complete! 2021-05-21T15:09:42.872 INFO:teuthology.provision.fog.gibba041:Deploy complete! 2021-05-21T15:09:43.040 INFO:teuthology.lock.ops:Checking gibba041.front.sepia.ceph.com 2021-05-21T15:09:43.041 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-21T15:09:43.060 INFO:teuthology.lock.ops:Checking gibba042.front.sepia.ceph.com 2021-05-21T15:09:43.060 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-21T15:09:43.077 INFO:teuthology.lock.ops:Updating [gibba041.front.sepia.ceph.com]: set os type and version on server 2021-05-21T15:09:43.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-21T15:09:43.326 INFO:teuthology.orchestra.run.gibba041.stdout:x86_64 2021-05-21T15:09:43.377 INFO:teuthology.orchestra.run.gibba041.stdout:NAME="CentOS Linux" 2021-05-21T15:09:43.378 INFO:teuthology.orchestra.run.gibba041.stdout:VERSION="8" 2021-05-21T15:09:43.378 INFO:teuthology.orchestra.run.gibba041.stdout:ID="centos" 2021-05-21T15:09:43.378 INFO:teuthology.orchestra.run.gibba041.stdout:ID_LIKE="rhel fedora" 2021-05-21T15:09:43.378 INFO:teuthology.orchestra.run.gibba041.stdout:VERSION_ID="8" 2021-05-21T15:09:43.379 INFO:teuthology.orchestra.run.gibba041.stdout:PLATFORM_ID="platform:el8" 2021-05-21T15:09:43.379 INFO:teuthology.orchestra.run.gibba041.stdout:PRETTY_NAME="CentOS Linux 8" 2021-05-21T15:09:43.379 INFO:teuthology.orchestra.run.gibba041.stdout:ANSI_COLOR="0;31" 2021-05-21T15:09:43.379 INFO:teuthology.orchestra.run.gibba041.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-21T15:09:43.379 INFO:teuthology.orchestra.run.gibba041.stdout:HOME_URL="https://centos.org/" 2021-05-21T15:09:43.380 INFO:teuthology.orchestra.run.gibba041.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-21T15:09:43.380 INFO:teuthology.orchestra.run.gibba041.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-21T15:09:43.380 INFO:teuthology.orchestra.run.gibba041.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-21T15:09:43.381 INFO:teuthology.lock.ops:Updating gibba041.front.sepia.ceph.com on lock server 2021-05-21T15:09:43.398 INFO:teuthology.lock.ops:Updating [gibba042.front.sepia.ceph.com]: set os type and version on server 2021-05-21T15:09:43.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-21T15:09:43.658 INFO:teuthology.orchestra.run.gibba042.stdout:x86_64 2021-05-21T15:09:43.710 INFO:teuthology.orchestra.run.gibba042.stdout:NAME="CentOS Linux" 2021-05-21T15:09:43.711 INFO:teuthology.orchestra.run.gibba042.stdout:VERSION="8" 2021-05-21T15:09:43.711 INFO:teuthology.orchestra.run.gibba042.stdout:ID="centos" 2021-05-21T15:09:43.711 INFO:teuthology.orchestra.run.gibba042.stdout:ID_LIKE="rhel fedora" 2021-05-21T15:09:43.711 INFO:teuthology.orchestra.run.gibba042.stdout:VERSION_ID="8" 2021-05-21T15:09:43.712 INFO:teuthology.orchestra.run.gibba042.stdout:PLATFORM_ID="platform:el8" 2021-05-21T15:09:43.712 INFO:teuthology.orchestra.run.gibba042.stdout:PRETTY_NAME="CentOS Linux 8" 2021-05-21T15:09:43.712 INFO:teuthology.orchestra.run.gibba042.stdout:ANSI_COLOR="0;31" 2021-05-21T15:09:43.712 INFO:teuthology.orchestra.run.gibba042.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-21T15:09:43.712 INFO:teuthology.orchestra.run.gibba042.stdout:HOME_URL="https://centos.org/" 2021-05-21T15:09:43.713 INFO:teuthology.orchestra.run.gibba042.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-21T15:09:43.713 INFO:teuthology.orchestra.run.gibba042.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-21T15:09:43.713 INFO:teuthology.orchestra.run.gibba042.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-21T15:09:43.714 INFO:teuthology.lock.ops:Updating gibba042.front.sepia.ceph.com on lock server 2021-05-21T15:09:43.809 INFO:teuthology.dispatcher.supervisor:Running job 6126464 2021-05-21T15:09:43.810 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_471a2f1efab907807e62c17be86bf6fb1a2f4522/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2021-05-21_12:26:30-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6126464 --name rfriedma-2021-05-21_12:26:30-rados-wip-ronenf-scrub-sched-distro-basic-gibba --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache} -- /home/teuthworker/archive/rfriedma-2021-05-21_12:26:30-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6126464/orig.config.yaml 2021-05-21T15:09:43.828 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2021-05-21_12:26:30-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6126464 2021-05-21T15:09:43.829 INFO:teuthology.dispatcher.supervisor:Job PID: 19818 2021-05-21T15:09:43.829 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-22T03:10:34.267 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-22T03:10:34.572 INFO:teuthology.kill:Killing Pids: {19818} 2021-05-22T03:10:36.535 INFO:teuthology.kill:Nuking machines: ['gibba041', 'gibba042'] 2021-05-22T03:10:37.533 INFO:teuthology.kill:2021-05-22 03:10:37,533.533 INFO:teuthology.nuke:targets: 2021-05-22T03:10:37.534 INFO:teuthology.kill: gibba041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvb+5hJuaC/H1nX6Wwb7wZiOJMOFtW4+vCYmU1Ogkxa+MXVtWYJ2zsnSrONrSIgZkR9G8qjHq+sgdSuFH8F4ukASR3H3E5CNQp/1b54vKNYbhUNXeM8cbxFLpHq4OyPklqA2xv+gwrtthxyZCUWS086mr5f4p61IGFH7C0GBJYz/zM5Eg2tmSRUDIBjNi6EMKbzRYW9YZ0UVnYfcwE8M1gHWUWFN73iHnZZrgPv7clyiWHybb2EmQiqZQLbCucRSxwwlUiAp0Yj8uuUqbCNMWOFFJIjgZroQDdeuBRi6fCD9giOD1H5B2fdQkItaxB2fY8/oRWh6ZfG+fgj44d2c3oBgYYisF6ljctV7NtPT1yxradiTzgmZNNj47+XE6niPSZo8DBGcvwHaFFEX4g8qdbUc9gl/jLz+clPAPsgKx7CXd340GsZZavuRB0V02srwx2KDvfxUeTrvMuS3Ax8k4JIB+dWy9J5h+50jqvzpefbHw3vRi9cpxDzbiCsrp0rRU= 2021-05-22T03:10:37.535 INFO:teuthology.kill: gibba042.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi2BYGP9PtwBtHeBRh6XvWmPU9ek9R6d4RMSIXm/53O/Ym4ShyXt6RJFy70t5QU9+D3DpEcaw/6qKL9M/l/ZqI7KFbPbWUDhXlk0jDW8/NP8wTNtUgJnntx4H7KFwaHMIdcjgb+ra+OCRxe37GYwwBRXb3sxyhK4/7O5z5UM0UOou5vtsXbzyoW/Yu5JWQ0/34aHPTMPWUu60E2n6dujOcv6VgMedFqoguNVZ9NFgETylauPa9DhflM7I0fAY+N3kqd/0HAhoId5Jlc3jz8r7LyX0fUjfR3BPBhOIi0e0AERU0GqNOkgYYSXq6026V4FIPYI+zLknZWI9b+dgpH5mOSo3OGjVrqe0CTx2K1rbwPUH+qLFcYQ063JYQQMa8RKwXZaVEeNT6pGBRAfSAP89uIBpJKr+HkuglxVhIL6fzODMSqQvq0oxVyOut/W3u7D5iXbqFnlarFhNJ3iYSl9qQ7hWMrgJWJeRL+ej6tHxYoxX2JJHdU44aeeQ3zFv1TpE= 2021-05-22T03:10:37.535 INFO:teuthology.kill:2021-05-22 03:10:37,534.534 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-22T03:10:37.537 INFO:teuthology.kill:2021-05-22 03:10:37,536.536 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-22T03:10:37.606 INFO:teuthology.kill:2021-05-22 03:10:37,606.606 INFO:teuthology.orchestra.console:Power off gibba042 2021-05-22T03:10:37.620 INFO:teuthology.kill:2021-05-22 03:10:37,619.619 INFO:teuthology.orchestra.console:Power off gibba041 2021-05-22T03:10:45.982 INFO:teuthology.kill:2021-05-22 03:10:45,982.982 INFO:teuthology.orchestra.console:Power off for gibba042 completed 2021-05-22T03:10:46.115 INFO:teuthology.kill:2021-05-22 03:10:46,114.114 INFO:teuthology.orchestra.console:Power off for gibba041 completed 2021-05-22T03:10:46.369 INFO:teuthology.task.internal:roles: ubuntu@gibba041.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-05-22T03:10:46.370 INFO:teuthology.task.internal:roles: ubuntu@gibba042.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-05-22T03:10:46.371 INFO:teuthology.misc:Compressing logs... 2021-05-22T03:10:46.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-22T03:11:46.435 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_471a2f1efab907807e62c17be86bf6fb1a2f4522/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_471a2f1efab907807e62c17be86bf6fb1a2f4522/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_471a2f1efab907807e62c17be86bf6fb1a2f4522/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_471a2f1efab907807e62c17be86bf6fb1a2f4522/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_471a2f1efab907807e62c17be86bf6fb1a2f4522/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_471a2f1efab907807e62c17be86bf6fb1a2f4522/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_471a2f1efab907807e62c17be86bf6fb1a2f4522/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-22T03:11:46.527 INFO:teuthology.kill:No teuthology processes running 2021-05-22T03:11:47.594 INFO:teuthology.kill:Nuking machines: ['gibba041', 'gibba042'] 2021-05-22T03:11:48.584 INFO:teuthology.kill:2021-05-22 03:11:48,584.584 INFO:teuthology.nuke:targets: 2021-05-22T03:11:48.585 INFO:teuthology.kill: gibba041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvb+5hJuaC/H1nX6Wwb7wZiOJMOFtW4+vCYmU1Ogkxa+MXVtWYJ2zsnSrONrSIgZkR9G8qjHq+sgdSuFH8F4ukASR3H3E5CNQp/1b54vKNYbhUNXeM8cbxFLpHq4OyPklqA2xv+gwrtthxyZCUWS086mr5f4p61IGFH7C0GBJYz/zM5Eg2tmSRUDIBjNi6EMKbzRYW9YZ0UVnYfcwE8M1gHWUWFN73iHnZZrgPv7clyiWHybb2EmQiqZQLbCucRSxwwlUiAp0Yj8uuUqbCNMWOFFJIjgZroQDdeuBRi6fCD9giOD1H5B2fdQkItaxB2fY8/oRWh6ZfG+fgj44d2c3oBgYYisF6ljctV7NtPT1yxradiTzgmZNNj47+XE6niPSZo8DBGcvwHaFFEX4g8qdbUc9gl/jLz+clPAPsgKx7CXd340GsZZavuRB0V02srwx2KDvfxUeTrvMuS3Ax8k4JIB+dWy9J5h+50jqvzpefbHw3vRi9cpxDzbiCsrp0rRU= 2021-05-22T03:11:48.585 INFO:teuthology.kill: gibba042.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi2BYGP9PtwBtHeBRh6XvWmPU9ek9R6d4RMSIXm/53O/Ym4ShyXt6RJFy70t5QU9+D3DpEcaw/6qKL9M/l/ZqI7KFbPbWUDhXlk0jDW8/NP8wTNtUgJnntx4H7KFwaHMIdcjgb+ra+OCRxe37GYwwBRXb3sxyhK4/7O5z5UM0UOou5vtsXbzyoW/Yu5JWQ0/34aHPTMPWUu60E2n6dujOcv6VgMedFqoguNVZ9NFgETylauPa9DhflM7I0fAY+N3kqd/0HAhoId5Jlc3jz8r7LyX0fUjfR3BPBhOIi0e0AERU0GqNOkgYYSXq6026V4FIPYI+zLknZWI9b+dgpH5mOSo3OGjVrqe0CTx2K1rbwPUH+qLFcYQ063JYQQMa8RKwXZaVEeNT6pGBRAfSAP89uIBpJKr+HkuglxVhIL6fzODMSqQvq0oxVyOut/W3u7D5iXbqFnlarFhNJ3iYSl9qQ7hWMrgJWJeRL+ej6tHxYoxX2JJHdU44aeeQ3zFv1TpE= 2021-05-22T03:11:48.605 INFO:teuthology.kill:2021-05-22 03:11:48,605.605 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-22T03:11:48.608 INFO:teuthology.kill:2021-05-22 03:11:48,608.608 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-22T03:11:48.676 INFO:teuthology.kill:2021-05-22 03:11:48,675.675 INFO:teuthology.orchestra.console:Power off gibba042 2021-05-22T03:11:48.689 INFO:teuthology.kill:2021-05-22 03:11:48,688.688 INFO:teuthology.orchestra.console:Power off gibba041 2021-05-22T03:11:48.888 INFO:teuthology.kill:2021-05-22 03:11:48,888.888 INFO:teuthology.orchestra.console:Power off for gibba042 completed 2021-05-22T03:11:48.915 INFO:teuthology.kill:2021-05-22 03:11:48,915.915 INFO:teuthology.orchestra.console:Power off for gibba041 completed 2021-05-22T03:11:49.055 INFO:teuthology.kill:2021-05-22 03:11:49,053.053 INFO:teuthology.lock.ops:unlocked gibba042.front.sepia.ceph.com 2021-05-22T03:11:49.078 INFO:teuthology.kill:2021-05-22 03:11:49,075.075 INFO:teuthology.lock.ops:unlocked gibba041.front.sepia.ceph.com 2021-05-22T03:13:49.415 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15