2021-08-04T09:56:09.965 INFO:root:teuthology version: 1.1.0-04c2febe 2021-08-04T09:56:10.075 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi053.front.sepia.ceph.com' reimaging 2021-08-04T09:56:10.076 INFO:teuthology.lock.ops:Updating [ubuntu@smithi053.front.sepia.ceph.com]: reset os type and version on server 2021-08-04T09:56:10.076 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2021-08-04T09:56:10.092 INFO:teuthology.lock.ops:Node 'ubuntu@smithi053.front.sepia.ceph.com' reimaging is complete 2021-08-04T09:56:10.092 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi087.front.sepia.ceph.com' reimaging 2021-08-04T09:56:10.092 INFO:teuthology.lock.ops:Updating [ubuntu@smithi087.front.sepia.ceph.com]: reset os type and version on server 2021-08-04T09:56:10.093 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2021-08-04T09:56:10.112 INFO:teuthology.lock.ops:Node 'ubuntu@smithi087.front.sepia.ceph.com' reimaging is complete 2021-08-04T09:56:10.312 INFO:teuthology.provision.fog.smithi053:Scheduling deploy of centos 8.2 2021-08-04T09:56:10.318 INFO:teuthology.provision.fog.smithi087:Scheduling deploy of centos 8.2 2021-08-04T09:56:10.695 INFO:teuthology.orchestra.console:Power off smithi053 2021-08-04T09:56:10.713 INFO:teuthology.orchestra.console:Power off smithi087 2021-08-04T09:56:19.070 INFO:teuthology.orchestra.console:Power off for smithi053 completed 2021-08-04T09:56:19.171 INFO:teuthology.orchestra.console:Power on smithi053 2021-08-04T09:56:23.192 INFO:teuthology.orchestra.console:Power off for smithi087 completed 2021-08-04T09:56:23.294 INFO:teuthology.orchestra.console:Power on smithi087 2021-08-04T09:56:27.545 INFO:teuthology.orchestra.console:Power on for smithi053 completed 2021-08-04T09:56:27.553 INFO:teuthology.orchestra.console:Power on for smithi087 completed 2021-08-04T09:56:27.648 INFO:teuthology.provision.fog.smithi053:Waiting for deploy to finish 2021-08-04T09:56:27.654 INFO:teuthology.provision.fog.smithi087:Waiting for deploy to finish 2021-08-04T10:00:36.759 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-04T10:00:37.076 INFO:teuthology.provision.fog.smithi053:Node is ready 2021-08-04T10:00:37.092 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053.front.sepia.ceph.com 2021-08-04T10:00:37.148 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53 smithi053.front.sepia.ceph.com smithi053 2021-08-04T10:00:37.477 INFO:teuthology.provision.fog.smithi053:Deploy complete! 2021-08-04T10:01:01.180 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-04T10:01:01.491 INFO:teuthology.provision.fog.smithi087:Node is ready 2021-08-04T10:01:01.505 INFO:teuthology.orchestra.run.smithi087.stdout:smithi087.front.sepia.ceph.com 2021-08-04T10:01:01.560 INFO:teuthology.orchestra.run.smithi087.stdout:172.21.15.87 smithi087.front.sepia.ceph.com smithi087 2021-08-04T10:01:01.884 INFO:teuthology.provision.fog.smithi087:Deploy complete! 2021-08-04T10:01:02.037 INFO:teuthology.lock.ops:Checking smithi053.front.sepia.ceph.com 2021-08-04T10:01:02.038 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-04T10:01:02.057 INFO:teuthology.lock.ops:Checking smithi087.front.sepia.ceph.com 2021-08-04T10:01:02.057 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-04T10:01:02.077 INFO:teuthology.lock.ops:Updating [smithi053.front.sepia.ceph.com]: set os type and version on server 2021-08-04T10:01:02.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-04T10:01:02.357 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2021-08-04T10:01:02.413 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Linux" 2021-08-04T10:01:02.414 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="8 (Core)" 2021-08-04T10:01:02.414 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2021-08-04T10:01:02.414 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2021-08-04T10:01:02.415 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="8" 2021-08-04T10:01:02.415 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el8" 2021-08-04T10:01:02.415 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-08-04T10:01:02.415 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2021-08-04T10:01:02.415 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-04T10:01:02.416 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://www.centos.org/" 2021-08-04T10:01:02.416 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-08-04T10:01:02.416 INFO:teuthology.orchestra.run.smithi053.stdout: 2021-08-04T10:01:02.416 INFO:teuthology.orchestra.run.smithi053.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-08-04T10:01:02.417 INFO:teuthology.orchestra.run.smithi053.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-08-04T10:01:02.417 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-08-04T10:01:02.417 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-08-04T10:01:02.417 INFO:teuthology.orchestra.run.smithi053.stdout: 2021-08-04T10:01:02.419 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2021-08-04T10:01:02.437 INFO:teuthology.lock.ops:Updating [smithi087.front.sepia.ceph.com]: set os type and version on server 2021-08-04T10:01:02.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-04T10:01:02.709 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2021-08-04T10:01:02.763 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="CentOS Linux" 2021-08-04T10:01:02.763 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="8 (Core)" 2021-08-04T10:01:02.764 INFO:teuthology.orchestra.run.smithi087.stdout:ID="centos" 2021-08-04T10:01:02.764 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE="rhel fedora" 2021-08-04T10:01:02.764 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="8" 2021-08-04T10:01:02.764 INFO:teuthology.orchestra.run.smithi087.stdout:PLATFORM_ID="platform:el8" 2021-08-04T10:01:02.765 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-08-04T10:01:02.765 INFO:teuthology.orchestra.run.smithi087.stdout:ANSI_COLOR="0;31" 2021-08-04T10:01:02.765 INFO:teuthology.orchestra.run.smithi087.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-04T10:01:02.765 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://www.centos.org/" 2021-08-04T10:01:02.766 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-08-04T10:01:02.766 INFO:teuthology.orchestra.run.smithi087.stdout: 2021-08-04T10:01:02.766 INFO:teuthology.orchestra.run.smithi087.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-08-04T10:01:02.767 INFO:teuthology.orchestra.run.smithi087.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-08-04T10:01:02.767 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-08-04T10:01:02.767 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-08-04T10:01:02.768 INFO:teuthology.orchestra.run.smithi087.stdout: 2021-08-04T10:01:02.768 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2021-08-04T10:01:02.891 INFO:teuthology.dispatcher.supervisor:Running job 6310875 2021-08-04T10:01:02.891 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2021-08-04_09:53:25-rados:cephadm:thrash-WIP_GBH_obj_scrubber2-distro-basic-smithi/6310875 --name benhanokh-2021-08-04_09:53:25-rados:cephadm:thrash-WIP_GBH_obj_scrubber2-distro-basic-smithi --description rados:cephadm:thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} -- /home/teuthworker/archive/benhanokh-2021-08-04_09:53:25-rados:cephadm:thrash-WIP_GBH_obj_scrubber2-distro-basic-smithi/6310875/orig.config.yaml 2021-08-04T10:01:02.897 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2021-08-04_09:53:25-rados:cephadm:thrash-WIP_GBH_obj_scrubber2-distro-basic-smithi/6310875 2021-08-04T10:01:02.898 INFO:teuthology.dispatcher.supervisor:Job PID: 15540 2021-08-04T10:01:02.898 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-08-04T22:01:53.201 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-08-04T22:01:53.417 INFO:teuthology.kill:Killing Pids: {15540} 2021-08-04T22:01:54.520 INFO:teuthology.kill:Nuking machines: ['smithi053', 'smithi087'] 2021-08-04T22:01:55.459 INFO:teuthology.kill:2021-08-04 22:01:55,459.459 INFO:teuthology.nuke:targets: 2021-08-04T22:01:55.460 INFO:teuthology.kill: smithi053.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKjFVc252UeDRVko69I0hrn5Um3g30TjaMo6Z4W7RifxRlVAroK9JVZPlGVALS7A1v5sQeYzn2eOmfFFt9FzCCSL2J16da9i1XtDazwB9lAWs3TCH7xJZ7v2oSV0jJ7vB5mtIhgQGjU8CxsafzDB3K49hlcq7z/YbFsKs1REomS+ulbK6Fsf7R+UVvzeyur0HstkWZNGCOQiCrCBYutOU8sIjEXD0ptta80rb6VH677lfEuidtEVavK6lNY6fvbUPljzdu+26ZJzS+DBs/ji5nN1rVPPQ1BkEXUmJuQSWwMwsKca4MnN0N4tchu2sIe9UQhfaD0rSxe2NQqW+mwsE2ls43W15lVHHhK1eLHE1Azgmue+JyQ4m37C5IOLNoNCTrPpNstklqUCXXOfwEKuF8Vs0n3Rmr0E+Yg66uYCZQA6AH1dv7lkAqZ+bFtZyFD/mJAVv8rZFplI1etnUwFM+k/Izdgpvxvqz1EO80j1+oI03+2+HJ1w4P/i8pT5Wbbes= 2021-08-04T22:01:55.460 INFO:teuthology.kill: smithi087.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSe5Ii9XTMLM7UYMuTwKIWPbL7YTKFGd/lhyZcw/YhkNDXA+yHhNaw14ID7DesZz1mu4OWZIr03W6ekcDnFxwFal0SDHmVajraOY4X6v/4dzfbjg797BLCh9A+m89T+K5DLUBBDLiMG4rG2douhY1mA+E8F7Vv7hqa2eJ6SL9gGww4N9MkJFfLjnKV/PqdTJO4yFalpoIRontfSXnDIrpjyOfSJtGsmvqtso9lUd8Mm+LVtdHDZNiHAj9LCaZUK7DY3+KWYa5QnXrHmwEUDWTvycZMpljbJMwErKt6LGF5d/Z+j3WEf/Jd/ONsKztlIv4Xjw8RbJSyGpOV0YQi/9ZtKjigbI30JMEgDDDxNJ8nvme+6T4Co6pkLmX8n9nImc8QoW28FwLelBZdUJzPHg4j/Hjtyo7QW9CLSxYCL2hqBASbzSMbLmbicNLsUBj4/LSF5IURssiVz8BTrPbLreyXhydnwbEXKnaRuT/GHBiYdcVMDkp7R5XqxScLCCt9DCM= 2021-08-04T22:01:55.460 INFO:teuthology.kill:2021-08-04 22:01:55,459.459 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-04T22:01:55.461 INFO:teuthology.kill:2021-08-04 22:01:55,461.461 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-04T22:01:55.519 INFO:teuthology.kill:2021-08-04 22:01:55,519.519 INFO:teuthology.orchestra.console:Power off smithi087 2021-08-04T22:01:55.532 INFO:teuthology.kill:2021-08-04 22:01:55,532.532 INFO:teuthology.orchestra.console:Power off smithi053 2021-08-04T22:02:03.892 INFO:teuthology.kill:2021-08-04 22:02:03,891.891 INFO:teuthology.orchestra.console:Power off for smithi053 completed 2021-08-04T22:02:07.997 INFO:teuthology.kill:2021-08-04 22:02:07,996.996 INFO:teuthology.orchestra.console:Power off for smithi087 completed 2021-08-04T22:02:08.212 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2021-08-04T22:02:08.213 INFO:teuthology.task.internal:roles: ubuntu@smithi087.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2021-08-04T22:02:08.213 INFO:teuthology.misc:Compressing logs... 2021-08-04T22:02:08.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-04T22:03:08.278 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_04c2febe7099917d97a71271f17abb5710030132/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-08-04T22:03:08.331 INFO:teuthology.kill:No teuthology processes running 2021-08-04T22:03:09.441 INFO:teuthology.kill:Nuking machines: ['smithi053', 'smithi087'] 2021-08-04T22:03:10.466 INFO:teuthology.kill:2021-08-04 22:03:10,466.466 INFO:teuthology.nuke:targets: 2021-08-04T22:03:10.467 INFO:teuthology.kill: smithi053.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKjFVc252UeDRVko69I0hrn5Um3g30TjaMo6Z4W7RifxRlVAroK9JVZPlGVALS7A1v5sQeYzn2eOmfFFt9FzCCSL2J16da9i1XtDazwB9lAWs3TCH7xJZ7v2oSV0jJ7vB5mtIhgQGjU8CxsafzDB3K49hlcq7z/YbFsKs1REomS+ulbK6Fsf7R+UVvzeyur0HstkWZNGCOQiCrCBYutOU8sIjEXD0ptta80rb6VH677lfEuidtEVavK6lNY6fvbUPljzdu+26ZJzS+DBs/ji5nN1rVPPQ1BkEXUmJuQSWwMwsKca4MnN0N4tchu2sIe9UQhfaD0rSxe2NQqW+mwsE2ls43W15lVHHhK1eLHE1Azgmue+JyQ4m37C5IOLNoNCTrPpNstklqUCXXOfwEKuF8Vs0n3Rmr0E+Yg66uYCZQA6AH1dv7lkAqZ+bFtZyFD/mJAVv8rZFplI1etnUwFM+k/Izdgpvxvqz1EO80j1+oI03+2+HJ1w4P/i8pT5Wbbes= 2021-08-04T22:03:10.467 INFO:teuthology.kill: smithi087.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSe5Ii9XTMLM7UYMuTwKIWPbL7YTKFGd/lhyZcw/YhkNDXA+yHhNaw14ID7DesZz1mu4OWZIr03W6ekcDnFxwFal0SDHmVajraOY4X6v/4dzfbjg797BLCh9A+m89T+K5DLUBBDLiMG4rG2douhY1mA+E8F7Vv7hqa2eJ6SL9gGww4N9MkJFfLjnKV/PqdTJO4yFalpoIRontfSXnDIrpjyOfSJtGsmvqtso9lUd8Mm+LVtdHDZNiHAj9LCaZUK7DY3+KWYa5QnXrHmwEUDWTvycZMpljbJMwErKt6LGF5d/Z+j3WEf/Jd/ONsKztlIv4Xjw8RbJSyGpOV0YQi/9ZtKjigbI30JMEgDDDxNJ8nvme+6T4Co6pkLmX8n9nImc8QoW28FwLelBZdUJzPHg4j/Hjtyo7QW9CLSxYCL2hqBASbzSMbLmbicNLsUBj4/LSF5IURssiVz8BTrPbLreyXhydnwbEXKnaRuT/GHBiYdcVMDkp7R5XqxScLCCt9DCM= 2021-08-04T22:03:10.487 INFO:teuthology.kill:2021-08-04 22:03:10,487.487 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-04T22:03:10.490 INFO:teuthology.kill:2021-08-04 22:03:10,490.490 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-04T22:03:10.546 INFO:teuthology.kill:2021-08-04 22:03:10,545.545 INFO:teuthology.orchestra.console:Power off smithi087 2021-08-04T22:03:10.558 INFO:teuthology.kill:2021-08-04 22:03:10,557.557 INFO:teuthology.orchestra.console:Power off smithi053 2021-08-04T22:03:10.687 INFO:teuthology.kill:2021-08-04 22:03:10,686.686 INFO:teuthology.orchestra.console:Power off for smithi087 completed 2021-08-04T22:03:10.699 INFO:teuthology.kill:2021-08-04 22:03:10,699.699 INFO:teuthology.orchestra.console:Power off for smithi053 completed 2021-08-04T22:03:10.827 INFO:teuthology.kill:2021-08-04 22:03:10,826.826 INFO:teuthology.lock.ops:unlocked smithi087.front.sepia.ceph.com 2021-08-04T22:03:10.837 INFO:teuthology.kill:2021-08-04 22:03:10,837.837 INFO:teuthology.lock.ops:unlocked smithi053.front.sepia.ceph.com 2021-08-04T22:05:11.127 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-08-04T22:05:11.216 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-08-04T22:05:11.216 INFO:teuthology.nuke:Checking targets against current locks 2021-08-04T22:05:11.320 INFO:teuthology.nuke:Not nuking smithi087.front.sepia.ceph.com because description doesn't match 2021-08-04T22:05:11.321 INFO:teuthology.nuke:Not nuking smithi053.front.sepia.ceph.com because description doesn't match