2021-02-22T16:02:57.877 INFO:root:teuthology version: 1.1.0-9a6b849c 2021-02-22T16:02:57.992 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi040.front.sepia.ceph.com' reimaging 2021-02-22T16:02:57.994 INFO:teuthology.lock.ops:Updating [ubuntu@smithi040.front.sepia.ceph.com]: reset os type and version on server 2021-02-22T16:02:57.995 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2021-02-22T16:02:58.012 INFO:teuthology.lock.ops:Node 'ubuntu@smithi040.front.sepia.ceph.com' reimaging is complete 2021-02-22T16:02:58.012 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi059.front.sepia.ceph.com' reimaging 2021-02-22T16:02:58.013 INFO:teuthology.lock.ops:Updating [ubuntu@smithi059.front.sepia.ceph.com]: reset os type and version on server 2021-02-22T16:02:58.014 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2021-02-22T16:02:58.030 INFO:teuthology.lock.ops:Node 'ubuntu@smithi059.front.sepia.ceph.com' reimaging is complete 2021-02-22T16:02:58.221 INFO:teuthology.provision.fog.smithi040:Scheduling deploy of centos 8.2 2021-02-22T16:02:58.243 INFO:teuthology.provision.fog.smithi059:Scheduling deploy of centos 8.2 2021-02-22T16:02:58.579 INFO:teuthology.orchestra.console:Power off smithi040 2021-02-22T16:02:58.630 INFO:teuthology.orchestra.console:Power off smithi059 2021-02-22T16:03:06.967 INFO:teuthology.orchestra.console:Power off for smithi040 completed 2021-02-22T16:03:07.001 INFO:teuthology.orchestra.console:Power off for smithi059 completed 2021-02-22T16:03:07.068 INFO:teuthology.orchestra.console:Power on smithi040 2021-02-22T16:03:07.111 INFO:teuthology.orchestra.console:Power on smithi059 2021-02-22T16:03:11.342 INFO:teuthology.orchestra.console:Power on for smithi040 completed 2021-02-22T16:03:11.444 INFO:teuthology.provision.fog.smithi040:Waiting for deploy to finish 2021-02-22T16:03:15.529 INFO:teuthology.orchestra.console:Power on for smithi059 completed 2021-02-22T16:03:15.630 INFO:teuthology.provision.fog.smithi059:Waiting for deploy to finish 2021-02-22T16:07:28.095 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-22T16:07:28.423 INFO:teuthology.provision.fog.smithi040:Node is ready 2021-02-22T16:07:28.438 INFO:teuthology.orchestra.run.smithi040.stdout:smithi040.front.sepia.ceph.com 2021-02-22T16:07:28.494 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40 smithi040.front.sepia.ceph.com smithi040 2021-02-22T16:07:28.829 INFO:teuthology.provision.fog.smithi040:Deploy complete! 2021-02-22T16:07:32.250 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-22T16:07:32.568 INFO:teuthology.provision.fog.smithi059:Node is ready 2021-02-22T16:07:32.584 INFO:teuthology.orchestra.run.smithi059.stdout:smithi059.front.sepia.ceph.com 2021-02-22T16:07:32.640 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.59 smithi059.front.sepia.ceph.com smithi059 2021-02-22T16:07:33.011 INFO:teuthology.provision.fog.smithi059:Deploy complete! 2021-02-22T16:07:33.144 INFO:teuthology.lock.ops:Checking smithi040.front.sepia.ceph.com 2021-02-22T16:07:33.144 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-22T16:07:33.160 INFO:teuthology.lock.ops:Checking smithi059.front.sepia.ceph.com 2021-02-22T16:07:33.161 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-22T16:07:33.175 INFO:teuthology.lock.ops:Updating [smithi040.front.sepia.ceph.com]: set os type and version on server 2021-02-22T16:07:33.176 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-22T16:07:33.459 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2021-02-22T16:07:33.513 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Linux" 2021-02-22T16:07:33.514 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="8 (Core)" 2021-02-22T16:07:33.514 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2021-02-22T16:07:33.514 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2021-02-22T16:07:33.514 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="8" 2021-02-22T16:07:33.515 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el8" 2021-02-22T16:07:33.515 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-02-22T16:07:33.515 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2021-02-22T16:07:33.515 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-02-22T16:07:33.515 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://www.centos.org/" 2021-02-22T16:07:33.516 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-02-22T16:07:33.516 INFO:teuthology.orchestra.run.smithi040.stdout: 2021-02-22T16:07:33.516 INFO:teuthology.orchestra.run.smithi040.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-02-22T16:07:33.516 INFO:teuthology.orchestra.run.smithi040.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-02-22T16:07:33.516 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-02-22T16:07:33.517 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-02-22T16:07:33.517 INFO:teuthology.orchestra.run.smithi040.stdout: 2021-02-22T16:07:33.518 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2021-02-22T16:07:33.533 INFO:teuthology.lock.ops:Updating [smithi059.front.sepia.ceph.com]: set os type and version on server 2021-02-22T16:07:33.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-22T16:07:33.808 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2021-02-22T16:07:33.863 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="CentOS Linux" 2021-02-22T16:07:33.863 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="8 (Core)" 2021-02-22T16:07:33.864 INFO:teuthology.orchestra.run.smithi059.stdout:ID="centos" 2021-02-22T16:07:33.864 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE="rhel fedora" 2021-02-22T16:07:33.864 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="8" 2021-02-22T16:07:33.864 INFO:teuthology.orchestra.run.smithi059.stdout:PLATFORM_ID="platform:el8" 2021-02-22T16:07:33.864 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-02-22T16:07:33.865 INFO:teuthology.orchestra.run.smithi059.stdout:ANSI_COLOR="0;31" 2021-02-22T16:07:33.865 INFO:teuthology.orchestra.run.smithi059.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-02-22T16:07:33.865 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://www.centos.org/" 2021-02-22T16:07:33.865 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-02-22T16:07:33.865 INFO:teuthology.orchestra.run.smithi059.stdout: 2021-02-22T16:07:33.866 INFO:teuthology.orchestra.run.smithi059.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-02-22T16:07:33.866 INFO:teuthology.orchestra.run.smithi059.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-02-22T16:07:33.866 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-02-22T16:07:33.866 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-02-22T16:07:33.866 INFO:teuthology.orchestra.run.smithi059.stdout: 2021-02-22T16:07:33.867 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2021-02-22T16:07:33.946 INFO:teuthology.dispatcher.supervisor:Running job 5901912 2021-02-22T16:07:33.946 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/virtualenv/bin/teuthology -v --owner scheduled_xiubli@teuthology --archive /home/teuthworker/archive/xiubli-2021-02-22_04:53:19-fs:functional-wip-lxb-testing-20210221-1527-distro-basic-smithi/5901912 --name xiubli-2021-02-22_04:53:19-fs:functional-wip-lxb-testing-20210221-1527-distro-basic-smithi --description fs:functional/{begin.yaml clusters/1a3s-mds-4c-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} distro/{centos_8.yaml} mount/fuse.yaml objectstore/bluestore-bitmap.yaml overrides/{frag_enable.yaml no_client_pidfile.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/client-readahead.yaml} -- /home/teuthworker/archive/xiubli-2021-02-22_04:53:19-fs:functional-wip-lxb-testing-20210221-1527-distro-basic-smithi/5901912/orig.config.yaml 2021-02-22T16:07:33.950 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/xiubli-2021-02-22_04:53:19-fs:functional-wip-lxb-testing-20210221-1527-distro-basic-smithi/5901912 2021-02-22T16:07:33.951 INFO:teuthology.dispatcher.supervisor:Job PID: 31989 2021-02-22T16:07:33.952 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-02-23T04:07:50.180 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-02-23T04:07:52.156 INFO:teuthology.kill:Killing Pids: {31989} 2021-02-23T04:07:53.269 INFO:teuthology.kill:Nuking machines: ['smithi040', 'smithi059'] 2021-02-23T04:07:54.031 INFO:teuthology.kill:2021-02-23 04:07:54,031.031 INFO:teuthology.nuke:targets: 2021-02-23T04:07:54.032 INFO:teuthology.kill: smithi040.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxgdnWfE6a7s3XhmDfvmZ9gJeQzqMIijGXu9Li4jZiGc79CxQ/3cLWsi3a3JpPxLZFuhQsZ+Wehnyj3AAbG2U9wsJvajUEnfXfon640Z8+KTRszUXuNXwotsxW5O2waaJsy4O4zbCkRqapz0ybLL7bBEttAjW7p/PJqaBRjeX6hJ3MhcnzRV6m3f1inbq5hyPbQwA4wH6/cB8lnIo6F7xXL7sErHJ2rC3j5OjmDYFZ72u6orPdQby/+FO1sdNGB/wfgzJ++DTQGIwXPwfKEWfy4aZQMd8NuI4KJWsHowGw2rDYxJRZVL7WeIoszrQgFupEi04A8ZVTiSzlZibAOggTAhhZJq04I5J70/gVbZRaeWIUTsTggc+7j1PyQlJeNn+NkbuFcnMqyyo0sNx00IkjVq4rj6Y2KY2CnrXa7hJscFfKiFvINB9Y+U2uQwWd9fiqtLPhvmRCsfgh7krCLFQEkMXG/ZgM/d4jY2OpkifIWkd9JOn5Rw3oPhKOvy5hde0= 2021-02-23T04:07:54.034 INFO:teuthology.kill: smithi059.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCAZ/yJ40d3ee6/PxkcWVUS7uTLq08+si1efo1RFcXkz9VSVJD+6Qqqmi+b3s4XPAiNF97RiLUhS7xOGzRs9yQV6AyFUb8OX+K8oeXmhxB1fJ+km/sw2LMZT4oPxvISGm/ulOR0vzix09aznMcywTfk7CnDjf2o9jp40xt8VyFrsTz2ip+Bi1f2vHu1UwzcjOuLpTuaPYzcqw5yWALPtHx/3rxciUcFbw33FPgHX+GhfCm6tfHIXW8E9IUUwVwJTsKEfLb1JtV2IKvt2vJ5LgB8flaigu79A91jMOQPt7lU4RadQgUSRpeVwr7w/0Uz3y3R1VKHHKPfmq3kzKY4XqF/vLnuCEHbkXCjhMNg9QTT35gbrvPZf+2n4PUT+FYtShzp+SoskRnWcxKreWBgjxKDp/Y2zs6HoyfoezzQ9gagmyBln6bDAIWoZYwlzhJsJ6cu45pSMo3un6cVwxZJDQlvums1C+XHfiYvJg69YP3A0ckwSi7nd8If8JLTvs39As= 2021-02-23T04:07:54.034 INFO:teuthology.kill:2021-02-23 04:07:54,032.032 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T04:07:54.037 INFO:teuthology.kill:2021-02-23 04:07:54,037.037 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T04:07:54.094 INFO:teuthology.kill:2021-02-23 04:07:54,093.093 INFO:teuthology.orchestra.console:Power off smithi059 2021-02-23T04:07:54.107 INFO:teuthology.kill:2021-02-23 04:07:54,106.106 INFO:teuthology.orchestra.console:Power off smithi040 2021-02-23T04:08:02.465 INFO:teuthology.kill:2021-02-23 04:08:02,465.465 INFO:teuthology.orchestra.console:Power off for smithi059 completed 2021-02-23T04:08:06.581 INFO:teuthology.kill:2021-02-23 04:08:06,581.581 INFO:teuthology.orchestra.console:Power off for smithi040 completed 2021-02-23T04:08:06.842 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2021-02-23T04:08:06.844 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2021-02-23T04:08:06.848 INFO:teuthology.misc:Compressing logs... 2021-02-23T04:08:06.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-23T04:08:41.423 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-02-23T04:08:41.491 INFO:teuthology.kill:No teuthology processes running 2021-02-23T04:08:42.528 INFO:teuthology.kill:Nuking machines: ['smithi040', 'smithi059'] 2021-02-23T04:08:43.349 INFO:teuthology.kill:2021-02-23 04:08:43,348.348 INFO:teuthology.nuke:targets: 2021-02-23T04:08:43.350 INFO:teuthology.kill: smithi040.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxgdnWfE6a7s3XhmDfvmZ9gJeQzqMIijGXu9Li4jZiGc79CxQ/3cLWsi3a3JpPxLZFuhQsZ+Wehnyj3AAbG2U9wsJvajUEnfXfon640Z8+KTRszUXuNXwotsxW5O2waaJsy4O4zbCkRqapz0ybLL7bBEttAjW7p/PJqaBRjeX6hJ3MhcnzRV6m3f1inbq5hyPbQwA4wH6/cB8lnIo6F7xXL7sErHJ2rC3j5OjmDYFZ72u6orPdQby/+FO1sdNGB/wfgzJ++DTQGIwXPwfKEWfy4aZQMd8NuI4KJWsHowGw2rDYxJRZVL7WeIoszrQgFupEi04A8ZVTiSzlZibAOggTAhhZJq04I5J70/gVbZRaeWIUTsTggc+7j1PyQlJeNn+NkbuFcnMqyyo0sNx00IkjVq4rj6Y2KY2CnrXa7hJscFfKiFvINB9Y+U2uQwWd9fiqtLPhvmRCsfgh7krCLFQEkMXG/ZgM/d4jY2OpkifIWkd9JOn5Rw3oPhKOvy5hde0= 2021-02-23T04:08:43.485 INFO:teuthology.kill: smithi059.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCAZ/yJ40d3ee6/PxkcWVUS7uTLq08+si1efo1RFcXkz9VSVJD+6Qqqmi+b3s4XPAiNF97RiLUhS7xOGzRs9yQV6AyFUb8OX+K8oeXmhxB1fJ+km/sw2LMZT4oPxvISGm/ulOR0vzix09aznMcywTfk7CnDjf2o9jp40xt8VyFrsTz2ip+Bi1f2vHu1UwzcjOuLpTuaPYzcqw5yWALPtHx/3rxciUcFbw33FPgHX+GhfCm6tfHIXW8E9IUUwVwJTsKEfLb1JtV2IKvt2vJ5LgB8flaigu79A91jMOQPt7lU4RadQgUSRpeVwr7w/0Uz3y3R1VKHHKPfmq3kzKY4XqF/vLnuCEHbkXCjhMNg9QTT35gbrvPZf+2n4PUT+FYtShzp+SoskRnWcxKreWBgjxKDp/Y2zs6HoyfoezzQ9gagmyBln6bDAIWoZYwlzhJsJ6cu45pSMo3un6cVwxZJDQlvums1C+XHfiYvJg69YP3A0ckwSi7nd8If8JLTvs39As= 2021-02-23T04:08:43.499 INFO:teuthology.kill:2021-02-23 04:08:43,376.376 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T04:08:43.500 INFO:teuthology.kill:2021-02-23 04:08:43,379.379 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-23T04:08:43.500 INFO:teuthology.kill:2021-02-23 04:08:43,438.438 INFO:teuthology.orchestra.console:Power off smithi059 2021-02-23T04:08:43.501 INFO:teuthology.kill:2021-02-23 04:08:43,455.455 INFO:teuthology.orchestra.console:Power off smithi040 2021-02-23T04:08:43.586 INFO:teuthology.kill:2021-02-23 04:08:43,585.585 INFO:teuthology.orchestra.console:Power off for smithi059 completed 2021-02-23T04:08:43.598 INFO:teuthology.kill:2021-02-23 04:08:43,598.598 INFO:teuthology.orchestra.console:Power off for smithi040 completed 2021-02-23T04:08:43.730 INFO:teuthology.kill:2021-02-23 04:08:43,729.729 INFO:teuthology.lock.ops:unlocked smithi059.front.sepia.ceph.com 2021-02-23T04:08:43.738 INFO:teuthology.kill:2021-02-23 04:08:43,738.738 INFO:teuthology.lock.ops:unlocked smithi040.front.sepia.ceph.com 2021-02-23T04:10:43.959 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-02-23T04:10:44.075 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-02-23T04:10:44.077 INFO:teuthology.nuke:Checking targets against current locks 2021-02-23T04:10:44.354 INFO:teuthology.nuke:Not nuking smithi059.front.sepia.ceph.com because description doesn't match 2021-02-23T04:10:44.354 INFO:teuthology.nuke:Not nuking smithi040.front.sepia.ceph.com because description doesn't match