2021-04-23T16:11:51.851 INFO:root:teuthology version: 1.1.0-2713a3cd 2021-04-23T16:11:51.951 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba014.front.sepia.ceph.com' reimaging 2021-04-23T16:11:51.952 INFO:teuthology.lock.ops:Updating [ubuntu@gibba014.front.sepia.ceph.com]: reset os type and version on server 2021-04-23T16:11:51.953 INFO:teuthology.lock.ops:Updating gibba014.front.sepia.ceph.com on lock server 2021-04-23T16:11:52.080 INFO:teuthology.lock.ops:Node 'ubuntu@gibba014.front.sepia.ceph.com' reimaging is complete 2021-04-23T16:11:52.080 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba017.front.sepia.ceph.com' reimaging 2021-04-23T16:11:52.080 INFO:teuthology.lock.ops:Updating [ubuntu@gibba017.front.sepia.ceph.com]: reset os type and version on server 2021-04-23T16:11:52.080 INFO:teuthology.lock.ops:Updating gibba017.front.sepia.ceph.com on lock server 2021-04-23T16:11:52.094 INFO:teuthology.lock.ops:Node 'ubuntu@gibba017.front.sepia.ceph.com' reimaging is complete 2021-04-23T16:11:52.286 INFO:teuthology.provision.fog.gibba014:Scheduling deploy of centos 8.3 2021-04-23T16:11:52.291 INFO:teuthology.provision.fog.gibba017:Scheduling deploy of centos 8.3 2021-04-23T16:11:52.583 INFO:teuthology.orchestra.console:Power off gibba014 2021-04-23T16:11:52.601 INFO:teuthology.orchestra.console:Power off gibba017 2021-04-23T16:12:00.989 INFO:teuthology.orchestra.console:Power off for gibba014 completed 2021-04-23T16:12:00.994 INFO:teuthology.orchestra.console:Power off for gibba017 completed 2021-04-23T16:12:01.090 INFO:teuthology.orchestra.console:Power on gibba014 2021-04-23T16:12:01.098 INFO:teuthology.orchestra.console:Power on gibba017 2021-04-23T16:12:09.469 INFO:teuthology.orchestra.console:Power on for gibba014 completed 2021-04-23T16:12:09.550 INFO:teuthology.orchestra.console:Power on for gibba017 completed 2021-04-23T16:12:09.570 INFO:teuthology.provision.fog.gibba014:Waiting for deploy to finish 2021-04-23T16:12:09.651 INFO:teuthology.provision.fog.gibba017:Waiting for deploy to finish 2021-04-23T16:16:35.231 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-23T16:16:35.261 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-23T16:16:35.515 INFO:teuthology.provision.fog.gibba014:Node is ready 2021-04-23T16:16:35.561 INFO:teuthology.provision.fog.gibba017:Node is ready 2021-04-23T16:16:35.569 INFO:teuthology.orchestra.run.gibba014.stdout:gibba014.front.sepia.ceph.com 2021-04-23T16:16:35.572 INFO:teuthology.orchestra.run.gibba017.stdout:gibba017.front.sepia.ceph.com 2021-04-23T16:16:35.621 INFO:teuthology.orchestra.run.gibba014.stdout:172.21.2.114 gibba014.front.sepia.ceph.com gibba014 2021-04-23T16:16:35.624 INFO:teuthology.orchestra.run.gibba017.stdout:172.21.2.117 gibba017.front.sepia.ceph.com gibba017 2021-04-23T16:16:35.906 INFO:teuthology.provision.fog.gibba017:Deploy complete! 2021-04-23T16:16:35.910 INFO:teuthology.provision.fog.gibba014:Deploy complete! 2021-04-23T16:16:36.043 INFO:teuthology.lock.ops:Checking gibba014.front.sepia.ceph.com 2021-04-23T16:16:36.044 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-23T16:16:36.061 INFO:teuthology.lock.ops:Checking gibba017.front.sepia.ceph.com 2021-04-23T16:16:36.061 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-23T16:16:36.076 INFO:teuthology.lock.ops:Updating [gibba014.front.sepia.ceph.com]: set os type and version on server 2021-04-23T16:16:36.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-23T16:16:36.328 INFO:teuthology.orchestra.run.gibba014.stdout:x86_64 2021-04-23T16:16:36.380 INFO:teuthology.orchestra.run.gibba014.stdout:NAME="CentOS Linux" 2021-04-23T16:16:36.380 INFO:teuthology.orchestra.run.gibba014.stdout:VERSION="8" 2021-04-23T16:16:36.380 INFO:teuthology.orchestra.run.gibba014.stdout:ID="centos" 2021-04-23T16:16:36.380 INFO:teuthology.orchestra.run.gibba014.stdout:ID_LIKE="rhel fedora" 2021-04-23T16:16:36.381 INFO:teuthology.orchestra.run.gibba014.stdout:VERSION_ID="8" 2021-04-23T16:16:36.381 INFO:teuthology.orchestra.run.gibba014.stdout:PLATFORM_ID="platform:el8" 2021-04-23T16:16:36.381 INFO:teuthology.orchestra.run.gibba014.stdout:PRETTY_NAME="CentOS Linux 8" 2021-04-23T16:16:36.381 INFO:teuthology.orchestra.run.gibba014.stdout:ANSI_COLOR="0;31" 2021-04-23T16:16:36.381 INFO:teuthology.orchestra.run.gibba014.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-04-23T16:16:36.381 INFO:teuthology.orchestra.run.gibba014.stdout:HOME_URL="https://centos.org/" 2021-04-23T16:16:36.381 INFO:teuthology.orchestra.run.gibba014.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-04-23T16:16:36.381 INFO:teuthology.orchestra.run.gibba014.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-04-23T16:16:36.381 INFO:teuthology.orchestra.run.gibba014.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-04-23T16:16:36.382 INFO:teuthology.lock.ops:Updating gibba014.front.sepia.ceph.com on lock server 2021-04-23T16:16:36.411 INFO:teuthology.lock.ops:Updating [gibba017.front.sepia.ceph.com]: set os type and version on server 2021-04-23T16:16:36.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-23T16:16:36.662 INFO:teuthology.orchestra.run.gibba017.stdout:x86_64 2021-04-23T16:16:36.713 INFO:teuthology.orchestra.run.gibba017.stdout:NAME="CentOS Linux" 2021-04-23T16:16:36.714 INFO:teuthology.orchestra.run.gibba017.stdout:VERSION="8" 2021-04-23T16:16:36.714 INFO:teuthology.orchestra.run.gibba017.stdout:ID="centos" 2021-04-23T16:16:36.714 INFO:teuthology.orchestra.run.gibba017.stdout:ID_LIKE="rhel fedora" 2021-04-23T16:16:36.714 INFO:teuthology.orchestra.run.gibba017.stdout:VERSION_ID="8" 2021-04-23T16:16:36.714 INFO:teuthology.orchestra.run.gibba017.stdout:PLATFORM_ID="platform:el8" 2021-04-23T16:16:36.714 INFO:teuthology.orchestra.run.gibba017.stdout:PRETTY_NAME="CentOS Linux 8" 2021-04-23T16:16:36.714 INFO:teuthology.orchestra.run.gibba017.stdout:ANSI_COLOR="0;31" 2021-04-23T16:16:36.714 INFO:teuthology.orchestra.run.gibba017.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-04-23T16:16:36.714 INFO:teuthology.orchestra.run.gibba017.stdout:HOME_URL="https://centos.org/" 2021-04-23T16:16:36.714 INFO:teuthology.orchestra.run.gibba017.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-04-23T16:16:36.714 INFO:teuthology.orchestra.run.gibba017.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-04-23T16:16:36.715 INFO:teuthology.orchestra.run.gibba017.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-04-23T16:16:36.715 INFO:teuthology.lock.ops:Updating gibba017.front.sepia.ceph.com on lock server 2021-04-23T16:16:36.790 INFO:teuthology.dispatcher.supervisor:Running job 6067881 2021-04-23T16:16:36.791 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2021-04-23_08:19:18-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6067881 --name rfriedma-2021-04-23_08:19:18-rados-wip-ronenf-scrub-sched-distro-basic-gibba --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps} -- /home/teuthworker/archive/rfriedma-2021-04-23_08:19:18-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6067881/orig.config.yaml 2021-04-23T16:16:36.798 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2021-04-23_08:19:18-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6067881 2021-04-23T16:16:36.798 INFO:teuthology.dispatcher.supervisor:Job PID: 19594 2021-04-23T16:16:36.799 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-04-24T04:17:13.612 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-04-24T04:17:13.829 INFO:teuthology.kill:Killing Pids: {19594} 2021-04-24T04:17:15.003 INFO:teuthology.kill:Nuking machines: ['gibba014', 'gibba017'] 2021-04-24T04:17:15.928 INFO:teuthology.kill:2021-04-24 04:17:15,927.927 INFO:teuthology.nuke:targets: 2021-04-24T04:17:15.928 INFO:teuthology.kill: gibba014.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD4eqO2enmNy2aWzzdNymUaBtq5D/BVXDYDR7hTq/BmjeAbMqYp5UlSD8W4nN6l9omg+yt8RkmSVGKVFH1iXx+gDSGFhNKtWAsJPYbEMHcc7QwimFu66MRrg1e9wuyALjL5ma6SvTwtCc0gf6z+i6F3C0WwKX9TFjOGrXVA3tihV9SHXN6lSFmauxXui4pecIwONms7A6n5IeTSQSIvh/ZqEfCDDyPZfe1PuzeoaiK+e6u3UYtLaU+u8Ui8qv/DIfZ1JHuXnyf45oBAk45E3ErF7GdKQNMU7+C7k6gCcouLAKFkIwOjvbUwVRyLAW8tDlWDA9/aZC5F04JbNdCdeMoc1yAHW1JwJCZCAMxyTztr9uCqgkzK733ufT8jZP1avaARARMN/m2zZ6kGXEwd1hNfR/QlzjxyXUvrAOr5BOLW4xX6fSBWL3HP7VnCIg66p6Ab1eAroZLGLE5CE/bEOIgNRezwCCK8nchlf970tvOBGOCFZ0oBWJYHQx++9xn2ah0= 2021-04-24T04:17:15.928 INFO:teuthology.kill: gibba017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1JxS2ASyc7SKgr6M0roIWNibIY4MrC3y1oUbcBXvaak5cqMk4nDHjDmouBIEJwkm4XecXX6/xAx94UBQdHIKWWY8IUwblH8wTTXsD60Lo91T1McWeWKa2qmkYAc7R6eZpt7MHwJywmq/5fKPrBVK2xV4LtCiiMQMko35O+g3IwEPD2pQ78sEqDjT92nc3Fsx6pwfhxPTWA+oByWJJXSRC1CO8webzhi8rPZBZ18Jx+n8suH5bmD65ZbRINgmX6/MY3Y0aO/t3VapSqpqyVLC1UNYmDU4KuzXX2UqHuZmttsYY7HazY/VvQSqZm78BUIGcKxv0P5hUjJbLdiYxi10d0WyFfZ5ZeKu0zFEBtvi9ZsjrrOaXzuvpa79q38r62t4kWcj22uoITiNFceGBnXGBt5zJz9HfUnKurKBn81aZD85N58ym0F3ZdyoU7NhyZqKQzhXkUD/eH3m3UqORhfuJn1OmR0Fu6yaAvLEFPaTW+iV0Kkt6DwkmhnDyLDCZFq8= 2021-04-24T04:17:15.928 INFO:teuthology.kill:2021-04-24 04:17:15,928.928 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-24T04:17:15.930 INFO:teuthology.kill:2021-04-24 04:17:15,930.930 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-24T04:17:16.000 INFO:teuthology.kill:2021-04-24 04:17:16,000.000 INFO:teuthology.orchestra.console:Power off gibba014 2021-04-24T04:17:16.015 INFO:teuthology.kill:2021-04-24 04:17:16,014.014 INFO:teuthology.orchestra.console:Power off gibba017 2021-04-24T04:17:24.374 INFO:teuthology.kill:2021-04-24 04:17:24,373.373 INFO:teuthology.orchestra.console:Power off for gibba014 completed 2021-04-24T04:17:24.382 INFO:teuthology.kill:2021-04-24 04:17:24,382.382 INFO:teuthology.orchestra.console:Power off for gibba017 completed 2021-04-24T04:17:24.623 INFO:teuthology.task.internal:roles: ubuntu@gibba014.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-04-24T04:17:24.623 INFO:teuthology.task.internal:roles: ubuntu@gibba017.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-04-24T04:17:24.623 INFO:teuthology.misc:Compressing logs... 2021-04-24T04:17:24.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-24T04:18:24.684 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-04-24T04:18:24.749 INFO:teuthology.kill:No teuthology processes running 2021-04-24T04:18:25.811 INFO:teuthology.kill:Nuking machines: ['gibba014', 'gibba017'] 2021-04-24T04:18:26.730 INFO:teuthology.kill:2021-04-24 04:18:26,730.730 INFO:teuthology.nuke:targets: 2021-04-24T04:18:26.731 INFO:teuthology.kill: gibba014.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD4eqO2enmNy2aWzzdNymUaBtq5D/BVXDYDR7hTq/BmjeAbMqYp5UlSD8W4nN6l9omg+yt8RkmSVGKVFH1iXx+gDSGFhNKtWAsJPYbEMHcc7QwimFu66MRrg1e9wuyALjL5ma6SvTwtCc0gf6z+i6F3C0WwKX9TFjOGrXVA3tihV9SHXN6lSFmauxXui4pecIwONms7A6n5IeTSQSIvh/ZqEfCDDyPZfe1PuzeoaiK+e6u3UYtLaU+u8Ui8qv/DIfZ1JHuXnyf45oBAk45E3ErF7GdKQNMU7+C7k6gCcouLAKFkIwOjvbUwVRyLAW8tDlWDA9/aZC5F04JbNdCdeMoc1yAHW1JwJCZCAMxyTztr9uCqgkzK733ufT8jZP1avaARARMN/m2zZ6kGXEwd1hNfR/QlzjxyXUvrAOr5BOLW4xX6fSBWL3HP7VnCIg66p6Ab1eAroZLGLE5CE/bEOIgNRezwCCK8nchlf970tvOBGOCFZ0oBWJYHQx++9xn2ah0= 2021-04-24T04:18:26.731 INFO:teuthology.kill: gibba017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1JxS2ASyc7SKgr6M0roIWNibIY4MrC3y1oUbcBXvaak5cqMk4nDHjDmouBIEJwkm4XecXX6/xAx94UBQdHIKWWY8IUwblH8wTTXsD60Lo91T1McWeWKa2qmkYAc7R6eZpt7MHwJywmq/5fKPrBVK2xV4LtCiiMQMko35O+g3IwEPD2pQ78sEqDjT92nc3Fsx6pwfhxPTWA+oByWJJXSRC1CO8webzhi8rPZBZ18Jx+n8suH5bmD65ZbRINgmX6/MY3Y0aO/t3VapSqpqyVLC1UNYmDU4KuzXX2UqHuZmttsYY7HazY/VvQSqZm78BUIGcKxv0P5hUjJbLdiYxi10d0WyFfZ5ZeKu0zFEBtvi9ZsjrrOaXzuvpa79q38r62t4kWcj22uoITiNFceGBnXGBt5zJz9HfUnKurKBn81aZD85N58ym0F3ZdyoU7NhyZqKQzhXkUD/eH3m3UqORhfuJn1OmR0Fu6yaAvLEFPaTW+iV0Kkt6DwkmhnDyLDCZFq8= 2021-04-24T04:18:26.755 INFO:teuthology.kill:2021-04-24 04:18:26,755.755 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-24T04:18:26.768 INFO:teuthology.kill:2021-04-24 04:18:26,767.767 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-24T04:18:26.835 INFO:teuthology.kill:2021-04-24 04:18:26,834.834 INFO:teuthology.orchestra.console:Power off gibba014 2021-04-24T04:18:26.848 INFO:teuthology.kill:2021-04-24 04:18:26,847.847 INFO:teuthology.orchestra.console:Power off gibba017 2021-04-24T04:18:26.982 INFO:teuthology.kill:2021-04-24 04:18:26,981.981 INFO:teuthology.orchestra.console:Power off for gibba014 completed 2021-04-24T04:18:26.995 INFO:teuthology.kill:2021-04-24 04:18:26,995.995 INFO:teuthology.orchestra.console:Power off for gibba017 completed 2021-04-24T04:18:27.132 INFO:teuthology.kill:2021-04-24 04:18:27,132.132 INFO:teuthology.lock.ops:unlocked gibba014.front.sepia.ceph.com 2021-04-24T04:18:27.150 INFO:teuthology.kill:2021-04-24 04:18:27,150.150 INFO:teuthology.lock.ops:unlocked gibba017.front.sepia.ceph.com 2021-04-24T04:20:27.444 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-04-24T04:20:27.557 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-04-24T04:20:27.557 INFO:teuthology.nuke:Checking targets against current locks 2021-04-24T04:20:27.642 INFO:teuthology.nuke:Not nuking gibba014.front.sepia.ceph.com because description doesn't match 2021-04-24T04:20:27.643 INFO:teuthology.nuke:Not nuking gibba017.front.sepia.ceph.com because description doesn't match