2021-07-26T12:22:45.354 INFO:root:teuthology version: 1.1.0-b6f3efab 2021-07-26T12:22:45.445 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi062.front.sepia.ceph.com' reimaging 2021-07-26T12:22:45.446 INFO:teuthology.lock.ops:Updating [ubuntu@smithi062.front.sepia.ceph.com]: reset os type and version on server 2021-07-26T12:22:45.446 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2021-07-26T12:22:45.461 INFO:teuthology.lock.ops:Node 'ubuntu@smithi062.front.sepia.ceph.com' reimaging is complete 2021-07-26T12:22:45.462 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi064.front.sepia.ceph.com' reimaging 2021-07-26T12:22:45.462 INFO:teuthology.lock.ops:Updating [ubuntu@smithi064.front.sepia.ceph.com]: reset os type and version on server 2021-07-26T12:22:45.463 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2021-07-26T12:22:45.482 INFO:teuthology.lock.ops:Node 'ubuntu@smithi064.front.sepia.ceph.com' reimaging is complete 2021-07-26T12:22:45.685 INFO:teuthology.provision.fog.smithi064:Scheduling deploy of centos 8.2 2021-07-26T12:22:45.717 INFO:teuthology.provision.fog.smithi062:Scheduling deploy of centos 8.2 2021-07-26T12:22:46.019 INFO:teuthology.orchestra.console:Power off smithi064 2021-07-26T12:22:46.070 INFO:teuthology.orchestra.console:Power off smithi062 2021-07-26T12:22:46.211 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2021-07-26T12:22:46.312 INFO:teuthology.orchestra.console:Power on smithi062 2021-07-26T12:22:54.669 INFO:teuthology.orchestra.console:Power on for smithi062 completed 2021-07-26T12:22:54.770 INFO:teuthology.provision.fog.smithi062:Waiting for deploy to finish 2021-07-26T12:22:58.493 INFO:teuthology.orchestra.console:Power off for smithi064 completed 2021-07-26T12:22:58.594 INFO:teuthology.orchestra.console:Power on smithi064 2021-07-26T12:23:06.959 INFO:teuthology.orchestra.console:Power on for smithi064 completed 2021-07-26T12:23:07.060 INFO:teuthology.provision.fog.smithi064:Waiting for deploy to finish 2021-07-26T12:26:58.620 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-07-26T12:27:19.012 INFO:teuthology.provision.fog.smithi062:Node is ready 2021-07-26T12:27:19.032 INFO:teuthology.orchestra.run.smithi062.stdout:smithi062.front.sepia.ceph.com 2021-07-26T12:27:19.093 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62 smithi062.front.sepia.ceph.com smithi062 2021-07-26T12:27:19.457 INFO:teuthology.provision.fog.smithi062:Deploy complete! 2021-07-26T12:27:45.420 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-07-26T12:27:45.736 INFO:teuthology.provision.fog.smithi064:Node is ready 2021-07-26T12:27:45.752 INFO:teuthology.orchestra.run.smithi064.stdout:smithi064.front.sepia.ceph.com 2021-07-26T12:27:45.808 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64 smithi064.front.sepia.ceph.com smithi064 2021-07-26T12:27:46.142 INFO:teuthology.provision.fog.smithi064:Deploy complete! 2021-07-26T12:27:46.260 INFO:teuthology.lock.ops:Checking smithi062.front.sepia.ceph.com 2021-07-26T12:27:46.260 INFO:teuthology.lock.ops:New key found. Updating... 2021-07-26T12:27:46.275 INFO:teuthology.lock.ops:Checking smithi064.front.sepia.ceph.com 2021-07-26T12:27:46.276 INFO:teuthology.lock.ops:New key found. Updating... 2021-07-26T12:27:46.288 INFO:teuthology.lock.ops:Updating [smithi062.front.sepia.ceph.com]: set os type and version on server 2021-07-26T12:27:46.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-26T12:27:46.567 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2021-07-26T12:27:46.621 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="CentOS Linux" 2021-07-26T12:27:46.621 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="8 (Core)" 2021-07-26T12:27:46.622 INFO:teuthology.orchestra.run.smithi062.stdout:ID="centos" 2021-07-26T12:27:46.622 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="rhel fedora" 2021-07-26T12:27:46.622 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="8" 2021-07-26T12:27:46.622 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el8" 2021-07-26T12:27:46.623 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-07-26T12:27:46.623 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31" 2021-07-26T12:27:46.623 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-07-26T12:27:46.623 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://www.centos.org/" 2021-07-26T12:27:46.623 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-07-26T12:27:46.624 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-26T12:27:46.624 INFO:teuthology.orchestra.run.smithi062.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-07-26T12:27:46.624 INFO:teuthology.orchestra.run.smithi062.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-07-26T12:27:46.624 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-07-26T12:27:46.624 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-07-26T12:27:46.625 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-07-26T12:27:46.626 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2021-07-26T12:27:46.641 INFO:teuthology.lock.ops:Updating [smithi064.front.sepia.ceph.com]: set os type and version on server 2021-07-26T12:27:46.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-26T12:27:46.906 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2021-07-26T12:27:46.960 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Linux" 2021-07-26T12:27:46.960 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="8 (Core)" 2021-07-26T12:27:46.960 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2021-07-26T12:27:46.960 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2021-07-26T12:27:46.961 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="8" 2021-07-26T12:27:46.961 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el8" 2021-07-26T12:27:46.961 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-07-26T12:27:46.961 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2021-07-26T12:27:46.961 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-07-26T12:27:46.962 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://www.centos.org/" 2021-07-26T12:27:46.962 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-07-26T12:27:46.962 INFO:teuthology.orchestra.run.smithi064.stdout: 2021-07-26T12:27:46.962 INFO:teuthology.orchestra.run.smithi064.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-07-26T12:27:46.962 INFO:teuthology.orchestra.run.smithi064.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-07-26T12:27:46.963 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-07-26T12:27:46.963 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-07-26T12:27:46.963 INFO:teuthology.orchestra.run.smithi064.stdout: 2021-07-26T12:27:46.963 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2021-07-26T12:27:47.028 INFO:teuthology.dispatcher.supervisor:Running job 6293455 2021-07-26T12:27:47.028 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2021-07-26_12:21:56-rados:cephadm:thrash-WIP_GBH_obj_scrubber2-distro-basic-smithi/6293455 --name benhanokh-2021-07-26_12:21:56-rados:cephadm:thrash-WIP_GBH_obj_scrubber2-distro-basic-smithi --description rados:cephadm:thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} -- /home/teuthworker/archive/benhanokh-2021-07-26_12:21:56-rados:cephadm:thrash-WIP_GBH_obj_scrubber2-distro-basic-smithi/6293455/orig.config.yaml 2021-07-26T12:27:47.031 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2021-07-26_12:21:56-rados:cephadm:thrash-WIP_GBH_obj_scrubber2-distro-basic-smithi/6293455 2021-07-26T12:27:47.032 INFO:teuthology.dispatcher.supervisor:Job PID: 21849 2021-07-26T12:27:47.032 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-07-27T00:28:38.147 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-07-27T00:28:38.354 INFO:teuthology.kill:Killing Pids: {21849} 2021-07-27T00:28:39.549 INFO:teuthology.kill:Nuking machines: ['smithi062', 'smithi064'] 2021-07-27T00:28:40.533 INFO:teuthology.kill:2021-07-27 00:28:40,532.532 INFO:teuthology.nuke:targets: 2021-07-27T00:28:40.534 INFO:teuthology.kill: smithi062.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHz3/NKFqgnn+mbZRomi5TQ2rZB+eZDcQLT3ogL0w+EzTQUzeZHGPg/UNOLsc+iTtnhjfK6wKm0FapthJCTpiIUzSmnZDNFiw3j5C/Hn/4kbU/ikX0bJDBy/YXKi7gHoj65M2u+L6zLHaea/hUWwO5Dzvzn9ZE+esW+8UydQHKRvMr6lz7ZM3NAHjUcP581o2+Y8KQScbvocj9CJe1ySJKJIplUUSzAIFou3B9F6MFUY5Q/M9FNcst6X8dyWv9Qd81RY0edD0RRrkQYqWJfhzHWOK1lmDUsr+cHf2EwwDynbfv80rDkJeVrsAahFdJH9aLiGCDoh8X1VT8ENEuDo+dwqUfcSZPs1Zenl7yuE8FLpxVgQ4rkk48PqFGj/AgdUsu+csAUBS9XVI/u0BLk4LMBjCPM/Bmfc/aYSIiCP90JxQ+Qdx1NMuibVbhQTB5gKaZTLFz0S/wclgcHYvYV4ir38j0e/v8Vpde7ZmeZlwUVTC7DLSEQ/X9SzFCn7WI80M= 2021-07-27T00:28:40.534 INFO:teuthology.kill: smithi064.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqxDXxjU09vEI0RAD/tnbsDFFNA9kiuwwAkQ9CMOImiSisnugGaMCG69YXVnju3MqclOOPxHr43wJKFf1q/kwPalrfvg7GJ918giE+ajDRIHiJGPhHjvZAdyBbSs1fiDOWrApvS8V/P42AkBFg24JTz9BJj8Lf38g+PH7RFZP3EEXAwhmY+WQvRA5Vku7xYvoII3RRWlJaEEOi/7fx3uigU2hQzE3kzWg5GnsQPIGBt/YYJwRH1p9TA+gwIDEqLI7Bmqik+2c96PEA0K1EOuBl3qmr2L/UySfNoarId8c9eJGNWpJspjT+c9qW6sCO3m45fbhere95co0a1OAg3zLHRmELKwE7KAaYFTcnF5kVCz8V0pETl2gac5+f1jbvt52Vq9+1Oxs2Wkx9ftSj7K46mNDUfenKIPv6y+eFz5f5vqNLlrXqCKIDWcY3LVf6KlIl5lyfzIqAYO5O8sEWCp8UnlFFnCZf0z+wH3Fs9mEzLyuynxVczwTcu+Flg045IxM= 2021-07-27T00:28:40.534 INFO:teuthology.kill:2021-07-27 00:28:40,533.533 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-07-27T00:28:40.535 INFO:teuthology.kill:2021-07-27 00:28:40,535.535 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-07-27T00:28:40.596 INFO:teuthology.kill:2021-07-27 00:28:40,596.596 INFO:teuthology.orchestra.console:Power off smithi064 2021-07-27T00:28:40.613 INFO:teuthology.kill:2021-07-27 00:28:40,613.613 INFO:teuthology.orchestra.console:Power off smithi062 2021-07-27T00:28:48.983 INFO:teuthology.kill:2021-07-27 00:28:48,981.981 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2021-07-27T00:28:53.087 INFO:teuthology.kill:2021-07-27 00:28:53,086.086 INFO:teuthology.orchestra.console:Power off for smithi064 completed 2021-07-27T00:28:53.345 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2021-07-27T00:28:53.345 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2021-07-27T00:28:53.346 INFO:teuthology.misc:Compressing logs... 2021-07-27T00:28:53.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-27T00:29:27.883 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-07-27T00:29:27.968 INFO:teuthology.kill:No teuthology processes running 2021-07-27T00:29:29.315 INFO:teuthology.kill:Nuking machines: ['smithi062', 'smithi064'] 2021-07-27T00:29:30.597 INFO:teuthology.kill:2021-07-27 00:29:30,597.597 INFO:teuthology.nuke:targets: 2021-07-27T00:29:30.598 INFO:teuthology.kill: smithi062.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHz3/NKFqgnn+mbZRomi5TQ2rZB+eZDcQLT3ogL0w+EzTQUzeZHGPg/UNOLsc+iTtnhjfK6wKm0FapthJCTpiIUzSmnZDNFiw3j5C/Hn/4kbU/ikX0bJDBy/YXKi7gHoj65M2u+L6zLHaea/hUWwO5Dzvzn9ZE+esW+8UydQHKRvMr6lz7ZM3NAHjUcP581o2+Y8KQScbvocj9CJe1ySJKJIplUUSzAIFou3B9F6MFUY5Q/M9FNcst6X8dyWv9Qd81RY0edD0RRrkQYqWJfhzHWOK1lmDUsr+cHf2EwwDynbfv80rDkJeVrsAahFdJH9aLiGCDoh8X1VT8ENEuDo+dwqUfcSZPs1Zenl7yuE8FLpxVgQ4rkk48PqFGj/AgdUsu+csAUBS9XVI/u0BLk4LMBjCPM/Bmfc/aYSIiCP90JxQ+Qdx1NMuibVbhQTB5gKaZTLFz0S/wclgcHYvYV4ir38j0e/v8Vpde7ZmeZlwUVTC7DLSEQ/X9SzFCn7WI80M= 2021-07-27T00:29:30.599 INFO:teuthology.kill: smithi064.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqxDXxjU09vEI0RAD/tnbsDFFNA9kiuwwAkQ9CMOImiSisnugGaMCG69YXVnju3MqclOOPxHr43wJKFf1q/kwPalrfvg7GJ918giE+ajDRIHiJGPhHjvZAdyBbSs1fiDOWrApvS8V/P42AkBFg24JTz9BJj8Lf38g+PH7RFZP3EEXAwhmY+WQvRA5Vku7xYvoII3RRWlJaEEOi/7fx3uigU2hQzE3kzWg5GnsQPIGBt/YYJwRH1p9TA+gwIDEqLI7Bmqik+2c96PEA0K1EOuBl3qmr2L/UySfNoarId8c9eJGNWpJspjT+c9qW6sCO3m45fbhere95co0a1OAg3zLHRmELKwE7KAaYFTcnF5kVCz8V0pETl2gac5+f1jbvt52Vq9+1Oxs2Wkx9ftSj7K46mNDUfenKIPv6y+eFz5f5vqNLlrXqCKIDWcY3LVf6KlIl5lyfzIqAYO5O8sEWCp8UnlFFnCZf0z+wH3Fs9mEzLyuynxVczwTcu+Flg045IxM= 2021-07-27T00:29:30.626 INFO:teuthology.kill:2021-07-27 00:29:30,626.626 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-07-27T00:29:30.630 INFO:teuthology.kill:2021-07-27 00:29:30,630.630 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-07-27T00:29:30.703 INFO:teuthology.kill:2021-07-27 00:29:30,703.703 INFO:teuthology.orchestra.console:Power off smithi064 2021-07-27T00:29:30.732 INFO:teuthology.kill:2021-07-27 00:29:30,731.731 INFO:teuthology.orchestra.console:Power off smithi062 2021-07-27T00:29:30.889 INFO:teuthology.kill:2021-07-27 00:29:30,885.885 INFO:teuthology.orchestra.console:Power off for smithi064 completed 2021-07-27T00:29:30.898 INFO:teuthology.kill:2021-07-27 00:29:30,897.897 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2021-07-27T00:29:31.040 INFO:teuthology.kill:2021-07-27 00:29:31,039.039 INFO:teuthology.lock.ops:unlocked smithi064.front.sepia.ceph.com 2021-07-27T00:29:31.046 INFO:teuthology.kill:2021-07-27 00:29:31,046.046 INFO:teuthology.lock.ops:unlocked smithi062.front.sepia.ceph.com 2021-07-27T00:31:31.386 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-07-27T00:31:31.466 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-07-27T00:31:31.467 INFO:teuthology.nuke:Checking targets against current locks 2021-07-27T00:31:31.539 INFO:teuthology.nuke:Not nuking smithi064.front.sepia.ceph.com because description doesn't match 2021-07-27T00:31:31.539 INFO:teuthology.nuke:Not nuking smithi062.front.sepia.ceph.com because description doesn't match