2021-08-04T12:01:54.609 INFO:root:teuthology version: 1.1.0-04c2febe 2021-08-04T12:01:55.766 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi046.front.sepia.ceph.com' reimaging 2021-08-04T12:01:55.767 INFO:teuthology.lock.ops:Updating [ubuntu@smithi046.front.sepia.ceph.com]: reset os type and version on server 2021-08-04T12:01:55.768 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2021-08-04T12:01:55.786 INFO:teuthology.lock.ops:Node 'ubuntu@smithi046.front.sepia.ceph.com' reimaging is complete 2021-08-04T12:01:55.787 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi168.front.sepia.ceph.com' reimaging 2021-08-04T12:01:55.787 INFO:teuthology.lock.ops:Updating [ubuntu@smithi168.front.sepia.ceph.com]: reset os type and version on server 2021-08-04T12:01:55.787 INFO:teuthology.lock.ops:Updating smithi168.front.sepia.ceph.com on lock server 2021-08-04T12:01:55.807 INFO:teuthology.lock.ops:Node 'ubuntu@smithi168.front.sepia.ceph.com' reimaging is complete 2021-08-04T12:01:56.005 INFO:teuthology.provision.fog.smithi046:Scheduling deploy of centos 8.2 2021-08-04T12:01:56.031 INFO:teuthology.provision.fog.smithi168:Scheduling deploy of centos 8.2 2021-08-04T12:01:56.380 INFO:teuthology.orchestra.console:Power off smithi046 2021-08-04T12:01:56.393 INFO:teuthology.orchestra.console:Power off smithi168 2021-08-04T12:02:04.764 INFO:teuthology.orchestra.console:Power off for smithi046 completed 2021-08-04T12:02:04.865 INFO:teuthology.orchestra.console:Power on smithi046 2021-08-04T12:02:08.868 INFO:teuthology.orchestra.console:Power off for smithi168 completed 2021-08-04T12:02:08.969 INFO:teuthology.orchestra.console:Power on smithi168 2021-08-04T12:02:13.230 INFO:teuthology.orchestra.console:Power on for smithi046 completed 2021-08-04T12:02:13.331 INFO:teuthology.provision.fog.smithi046:Waiting for deploy to finish 2021-08-04T12:02:17.395 INFO:teuthology.orchestra.console:Power on for smithi168 completed 2021-08-04T12:02:17.496 INFO:teuthology.provision.fog.smithi168:Waiting for deploy to finish 2021-08-04T12:06:44.085 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-04T12:06:44.405 INFO:teuthology.provision.fog.smithi046:Node is ready 2021-08-04T12:06:44.420 INFO:teuthology.orchestra.run.smithi046.stdout:smithi046.front.sepia.ceph.com 2021-08-04T12:06:44.491 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46 smithi046.front.sepia.ceph.com smithi046 2021-08-04T12:06:44.866 INFO:teuthology.provision.fog.smithi046:Deploy complete! 2021-08-04T12:07:06.798 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-04T12:07:07.093 INFO:teuthology.provision.fog.smithi168:Node is ready 2021-08-04T12:07:07.108 INFO:teuthology.orchestra.run.smithi168.stdout:smithi168.front.sepia.ceph.com 2021-08-04T12:07:07.163 INFO:teuthology.orchestra.run.smithi168.stdout:172.21.15.168 smithi168.front.sepia.ceph.com smithi168 2021-08-04T12:07:07.488 INFO:teuthology.provision.fog.smithi168:Deploy complete! 2021-08-04T12:07:07.629 INFO:teuthology.lock.ops:Checking smithi046.front.sepia.ceph.com 2021-08-04T12:07:07.631 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-04T12:07:07.649 INFO:teuthology.lock.ops:Checking smithi168.front.sepia.ceph.com 2021-08-04T12:07:07.650 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-04T12:07:07.668 INFO:teuthology.lock.ops:Updating [smithi046.front.sepia.ceph.com]: set os type and version on server 2021-08-04T12:07:07.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-04T12:07:07.889 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2021-08-04T12:07:07.945 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="CentOS Linux" 2021-08-04T12:07:07.946 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="8 (Core)" 2021-08-04T12:07:07.946 INFO:teuthology.orchestra.run.smithi046.stdout:ID="centos" 2021-08-04T12:07:07.946 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="rhel fedora" 2021-08-04T12:07:07.946 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="8" 2021-08-04T12:07:07.947 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el8" 2021-08-04T12:07:07.947 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-08-04T12:07:07.947 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2021-08-04T12:07:07.947 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-04T12:07:07.948 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://www.centos.org/" 2021-08-04T12:07:07.948 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-08-04T12:07:07.948 INFO:teuthology.orchestra.run.smithi046.stdout: 2021-08-04T12:07:07.948 INFO:teuthology.orchestra.run.smithi046.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-08-04T12:07:07.949 INFO:teuthology.orchestra.run.smithi046.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-08-04T12:07:07.949 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-08-04T12:07:07.949 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-08-04T12:07:07.949 INFO:teuthology.orchestra.run.smithi046.stdout: 2021-08-04T12:07:07.951 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2021-08-04T12:07:07.965 INFO:teuthology.lock.ops:Updating [smithi168.front.sepia.ceph.com]: set os type and version on server 2021-08-04T12:07:07.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-04T12:07:08.227 INFO:teuthology.orchestra.run.smithi168.stdout:x86_64 2021-08-04T12:07:08.281 INFO:teuthology.orchestra.run.smithi168.stdout:NAME="CentOS Linux" 2021-08-04T12:07:08.281 INFO:teuthology.orchestra.run.smithi168.stdout:VERSION="8 (Core)" 2021-08-04T12:07:08.281 INFO:teuthology.orchestra.run.smithi168.stdout:ID="centos" 2021-08-04T12:07:08.282 INFO:teuthology.orchestra.run.smithi168.stdout:ID_LIKE="rhel fedora" 2021-08-04T12:07:08.282 INFO:teuthology.orchestra.run.smithi168.stdout:VERSION_ID="8" 2021-08-04T12:07:08.283 INFO:teuthology.orchestra.run.smithi168.stdout:PLATFORM_ID="platform:el8" 2021-08-04T12:07:08.283 INFO:teuthology.orchestra.run.smithi168.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-08-04T12:07:08.284 INFO:teuthology.orchestra.run.smithi168.stdout:ANSI_COLOR="0;31" 2021-08-04T12:07:08.284 INFO:teuthology.orchestra.run.smithi168.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-04T12:07:08.285 INFO:teuthology.orchestra.run.smithi168.stdout:HOME_URL="https://www.centos.org/" 2021-08-04T12:07:08.285 INFO:teuthology.orchestra.run.smithi168.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-08-04T12:07:08.286 INFO:teuthology.orchestra.run.smithi168.stdout: 2021-08-04T12:07:08.286 INFO:teuthology.orchestra.run.smithi168.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-08-04T12:07:08.286 INFO:teuthology.orchestra.run.smithi168.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-08-04T12:07:08.287 INFO:teuthology.orchestra.run.smithi168.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-08-04T12:07:08.287 INFO:teuthology.orchestra.run.smithi168.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-08-04T12:07:08.287 INFO:teuthology.orchestra.run.smithi168.stdout: 2021-08-04T12:07:08.288 INFO:teuthology.lock.ops:Updating smithi168.front.sepia.ceph.com on lock server 2021-08-04T12:07:08.359 INFO:teuthology.dispatcher.supervisor:Running job 6310925 2021-08-04T12:07:08.360 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2021-08-04_12:01:03-rados:cephadm-WIP_GBH_obj_scrubber2-distro-basic-smithi/6310925 --name benhanokh-2021-08-04_12:01:03-rados:cephadm-WIP_GBH_obj_scrubber2-distro-basic-smithi --description rados:cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} -- /home/teuthworker/archive/benhanokh-2021-08-04_12:01:03-rados:cephadm-WIP_GBH_obj_scrubber2-distro-basic-smithi/6310925/orig.config.yaml 2021-08-04T12:07:08.366 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2021-08-04_12:01:03-rados:cephadm-WIP_GBH_obj_scrubber2-distro-basic-smithi/6310925 2021-08-04T12:07:08.367 INFO:teuthology.dispatcher.supervisor:Job PID: 18984 2021-08-04T12:07:08.368 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-08-05T00:07:56.173 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-08-05T00:07:56.375 INFO:teuthology.kill:Killing Pids: {18984} 2021-08-05T00:07:57.355 INFO:teuthology.kill:Nuking machines: ['smithi046', 'smithi168'] 2021-08-05T00:07:58.231 INFO:teuthology.kill:2021-08-05 00:07:58,230.230 INFO:teuthology.nuke:targets: 2021-08-05T00:07:58.231 INFO:teuthology.kill: smithi046.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtKU/wFKY4gwvP58DVS+zv63XlC4EutRlJJGBiIgud5OM9RoU81aBrv3QSML0ajvAiiMeYG9Lith6m/zECWNBUeFiWbj7qmiCBOlJGKDHl5UNEQqb893cFM49ORPjMaXFqFWPp+/ivhPS1uoNZabvY9jpnpF6PPOSMTrBPWJLuz9Zy3krvN5gxDEA5PKQ1Aj5j4FeiW06PnN+fa/RS0JjOLWEmWIwiqD73C8CEW8PKu7AHl6okhFSwcKhR0op1wIQtBMBNW9bAZszMLSAYfX0sjjejWeEkHqdvtoXhA3HBalUH8jj3akIz9knrkdIjVJ8qjvVagsr1OyZ/eyUfkkRW2uQpaBQYxzPOKBUCabho3tcXPR03h3hFquTX8KIn714Jh2xaJc+bEoSzfZCWagkInptjLwpx2HGEgQVLegSoQhR1JgcerHaHCI+A80gzVOujxZRb7J2wrILNffUtV3U+r6pW4VfJZBOtn1oJFAq7qdnO3afMoKkd3PQdxGGJ3Z8= 2021-08-05T00:07:58.232 INFO:teuthology.kill: smithi168.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBdbyGshhiM6pwgv7kcAaKz5IsfuH9uovq8es3tRtEMj4ikFoHgFyxasbA41Qs+8mAasdrnINKPEdmdZ2cUkt/vrvRKY7fnf0Aawc7kBXd2IdD2YiRLb/8dwhsY1fTYJcsntQKG50w9dtP2/uhr7+dRyRCBJsE0tGTTT0H2RPAD/ol9plkCeDdPj6OsOK34JW7RB3t3AM0CRlkLmeQtdIHupwLoJE+s0xsOxGcc/7y2DxujkGh8xT9NZfRxQPkHu9TiPZ2lyRGg+Csv65eA+7lfn1cii/cfYb2suU3koQR6F9Pf6VAQxAnDqaWB5W4OoLCxJxeksnho7THtHTtKT/Sw9NnK+ngt29qvW+XPaNo239+AXj+35+IYZ3418qxJvBLeVZvxWq9EfUWUmGANgCTL5BX5rKBQ+chKSdKrsWFgWCEKE4IhJbJH+RjoM9gSC28nPR0xEjf2KSsdOhTJrbxRUGPWEH+yYrKD9zw0R/+BnlZ9dgfzNWD5Ng5nXcDgM0= 2021-08-05T00:07:58.232 INFO:teuthology.kill:2021-08-05 00:07:58,231.231 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-05T00:07:58.233 INFO:teuthology.kill:2021-08-05 00:07:58,233.233 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-05T00:07:58.289 INFO:teuthology.kill:2021-08-05 00:07:58,289.289 INFO:teuthology.orchestra.console:Power off smithi168 2021-08-05T00:07:58.302 INFO:teuthology.kill:2021-08-05 00:07:58,302.302 INFO:teuthology.orchestra.console:Power off smithi046 2021-08-05T00:08:06.664 INFO:teuthology.kill:2021-08-05 00:08:06,664.664 INFO:teuthology.orchestra.console:Power off for smithi168 completed 2021-08-05T00:08:10.780 INFO:teuthology.kill:2021-08-05 00:08:10,780.780 INFO:teuthology.orchestra.console:Power off for smithi046 completed 2021-08-05T00:08:10.977 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2021-08-05T00:08:10.978 INFO:teuthology.task.internal:roles: ubuntu@smithi168.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2021-08-05T00:08:10.978 INFO:teuthology.misc:Compressing logs... 2021-08-05T00:08:10.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-05T00:08:45.515 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-08-05T00:08:45.560 INFO:teuthology.kill:No teuthology processes running 2021-08-05T00:08:46.509 INFO:teuthology.kill:Nuking machines: ['smithi046', 'smithi168'] 2021-08-05T00:08:47.395 INFO:teuthology.kill:2021-08-05 00:08:47,395.395 INFO:teuthology.nuke:targets: 2021-08-05T00:08:47.396 INFO:teuthology.kill: smithi046.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtKU/wFKY4gwvP58DVS+zv63XlC4EutRlJJGBiIgud5OM9RoU81aBrv3QSML0ajvAiiMeYG9Lith6m/zECWNBUeFiWbj7qmiCBOlJGKDHl5UNEQqb893cFM49ORPjMaXFqFWPp+/ivhPS1uoNZabvY9jpnpF6PPOSMTrBPWJLuz9Zy3krvN5gxDEA5PKQ1Aj5j4FeiW06PnN+fa/RS0JjOLWEmWIwiqD73C8CEW8PKu7AHl6okhFSwcKhR0op1wIQtBMBNW9bAZszMLSAYfX0sjjejWeEkHqdvtoXhA3HBalUH8jj3akIz9knrkdIjVJ8qjvVagsr1OyZ/eyUfkkRW2uQpaBQYxzPOKBUCabho3tcXPR03h3hFquTX8KIn714Jh2xaJc+bEoSzfZCWagkInptjLwpx2HGEgQVLegSoQhR1JgcerHaHCI+A80gzVOujxZRb7J2wrILNffUtV3U+r6pW4VfJZBOtn1oJFAq7qdnO3afMoKkd3PQdxGGJ3Z8= 2021-08-05T00:08:47.396 INFO:teuthology.kill: smithi168.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBdbyGshhiM6pwgv7kcAaKz5IsfuH9uovq8es3tRtEMj4ikFoHgFyxasbA41Qs+8mAasdrnINKPEdmdZ2cUkt/vrvRKY7fnf0Aawc7kBXd2IdD2YiRLb/8dwhsY1fTYJcsntQKG50w9dtP2/uhr7+dRyRCBJsE0tGTTT0H2RPAD/ol9plkCeDdPj6OsOK34JW7RB3t3AM0CRlkLmeQtdIHupwLoJE+s0xsOxGcc/7y2DxujkGh8xT9NZfRxQPkHu9TiPZ2lyRGg+Csv65eA+7lfn1cii/cfYb2suU3koQR6F9Pf6VAQxAnDqaWB5W4OoLCxJxeksnho7THtHTtKT/Sw9NnK+ngt29qvW+XPaNo239+AXj+35+IYZ3418qxJvBLeVZvxWq9EfUWUmGANgCTL5BX5rKBQ+chKSdKrsWFgWCEKE4IhJbJH+RjoM9gSC28nPR0xEjf2KSsdOhTJrbxRUGPWEH+yYrKD9zw0R/+BnlZ9dgfzNWD5Ng5nXcDgM0= 2021-08-05T00:08:47.414 INFO:teuthology.kill:2021-08-05 00:08:47,413.413 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-05T00:08:47.416 INFO:teuthology.kill:2021-08-05 00:08:47,416.416 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-05T00:08:47.470 INFO:teuthology.kill:2021-08-05 00:08:47,470.470 INFO:teuthology.orchestra.console:Power off smithi046 2021-08-05T00:08:47.482 INFO:teuthology.kill:2021-08-05 00:08:47,482.482 INFO:teuthology.orchestra.console:Power off smithi168 2021-08-05T00:08:47.610 INFO:teuthology.kill:2021-08-05 00:08:47,610.610 INFO:teuthology.orchestra.console:Power off for smithi046 completed 2021-08-05T00:08:47.619 INFO:teuthology.kill:2021-08-05 00:08:47,619.619 INFO:teuthology.orchestra.console:Power off for smithi168 completed 2021-08-05T00:08:47.749 INFO:teuthology.kill:2021-08-05 00:08:47,748.748 INFO:teuthology.lock.ops:unlocked smithi046.front.sepia.ceph.com 2021-08-05T00:08:47.757 INFO:teuthology.kill:2021-08-05 00:08:47,757.757 INFO:teuthology.lock.ops:unlocked smithi168.front.sepia.ceph.com 2021-08-05T00:10:48.023 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15