2021-02-25T20:48:09.867 INFO:root:teuthology version: 1.1.0-9a6b849c 2021-02-25T20:48:09.991 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi039.front.sepia.ceph.com' reimaging 2021-02-25T20:48:09.992 INFO:teuthology.lock.ops:Updating [ubuntu@smithi039.front.sepia.ceph.com]: reset os type and version on server 2021-02-25T20:48:09.992 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2021-02-25T20:48:10.012 INFO:teuthology.lock.ops:Node 'ubuntu@smithi039.front.sepia.ceph.com' reimaging is complete 2021-02-25T20:48:10.012 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi073.front.sepia.ceph.com' reimaging 2021-02-25T20:48:10.013 INFO:teuthology.lock.ops:Updating [ubuntu@smithi073.front.sepia.ceph.com]: reset os type and version on server 2021-02-25T20:48:10.014 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server 2021-02-25T20:48:10.031 INFO:teuthology.lock.ops:Node 'ubuntu@smithi073.front.sepia.ceph.com' reimaging is complete 2021-02-25T20:48:10.230 INFO:teuthology.provision.fog.smithi039:Scheduling deploy of rhel 8.3 2021-02-25T20:48:10.237 INFO:teuthology.provision.fog.smithi073:Scheduling deploy of rhel 8.3 2021-02-25T20:48:10.551 INFO:teuthology.orchestra.console:Power off smithi039 2021-02-25T20:48:10.610 INFO:teuthology.orchestra.console:Power off smithi073 2021-02-25T20:48:18.997 INFO:teuthology.orchestra.console:Power off for smithi039 completed 2021-02-25T20:48:19.103 INFO:teuthology.orchestra.console:Power on smithi039 2021-02-25T20:48:23.126 INFO:teuthology.orchestra.console:Power off for smithi073 completed 2021-02-25T20:48:23.229 INFO:teuthology.orchestra.console:Power on smithi073 2021-02-25T20:48:27.508 INFO:teuthology.orchestra.console:Power on for smithi073 completed 2021-02-25T20:48:27.543 INFO:teuthology.orchestra.console:Power on for smithi039 completed 2021-02-25T20:48:27.620 INFO:teuthology.provision.fog.smithi073:Waiting for deploy to finish 2021-02-25T20:48:27.644 INFO:teuthology.provision.fog.smithi039:Waiting for deploy to finish 2021-02-25T20:53:08.404 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-25T20:53:08.703 INFO:teuthology.provision.fog.smithi073:Node is ready 2021-02-25T20:53:08.718 INFO:teuthology.orchestra.run.smithi073.stdout:smithi073.front.sepia.ceph.com 2021-02-25T20:53:08.773 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73 smithi073.front.sepia.ceph.com smithi073 2021-02-25T20:53:09.092 INFO:teuthology.provision.fog.smithi073:Deploy complete! 2021-02-25T20:53:47.601 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-02-25T20:53:47.886 INFO:teuthology.provision.fog.smithi039:Node is ready 2021-02-25T20:53:47.901 INFO:teuthology.orchestra.run.smithi039.stdout:smithi039.front.sepia.ceph.com 2021-02-25T20:53:47.957 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39 smithi039.front.sepia.ceph.com smithi039 2021-02-25T20:53:48.292 INFO:teuthology.provision.fog.smithi039:Deploy complete! 2021-02-25T20:53:48.430 INFO:teuthology.lock.ops:Checking smithi039.front.sepia.ceph.com 2021-02-25T20:53:48.431 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-25T20:53:48.451 INFO:teuthology.lock.ops:Checking smithi073.front.sepia.ceph.com 2021-02-25T20:53:48.451 INFO:teuthology.lock.ops:New key found. Updating... 2021-02-25T20:53:48.467 INFO:teuthology.lock.ops:Updating [smithi039.front.sepia.ceph.com]: set os type and version on server 2021-02-25T20:53:48.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-25T20:53:48.773 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2021-02-25T20:53:48.842 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="Red Hat Enterprise Linux" 2021-02-25T20:53:48.842 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="8.3 (Ootpa)" 2021-02-25T20:53:48.842 INFO:teuthology.orchestra.run.smithi039.stdout:ID="rhel" 2021-02-25T20:53:48.843 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="fedora" 2021-02-25T20:53:48.843 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="8.3" 2021-02-25T20:53:48.843 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el8" 2021-02-25T20:53:48.843 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-02-25T20:53:48.843 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31" 2021-02-25T20:53:48.844 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-02-25T20:53:48.844 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://www.redhat.com/" 2021-02-25T20:53:48.844 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-02-25T20:53:48.844 INFO:teuthology.orchestra.run.smithi039.stdout: 2021-02-25T20:53:48.844 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-02-25T20:53:48.845 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-02-25T20:53:48.845 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-02-25T20:53:48.845 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-02-25T20:53:48.847 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2021-02-25T20:53:48.871 INFO:teuthology.lock.ops:Updating [smithi073.front.sepia.ceph.com]: set os type and version on server 2021-02-25T20:53:48.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-25T20:53:49.211 INFO:teuthology.orchestra.run.smithi073.stdout:x86_64 2021-02-25T20:53:49.265 INFO:teuthology.orchestra.run.smithi073.stdout:NAME="Red Hat Enterprise Linux" 2021-02-25T20:53:49.266 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION="8.3 (Ootpa)" 2021-02-25T20:53:49.266 INFO:teuthology.orchestra.run.smithi073.stdout:ID="rhel" 2021-02-25T20:53:49.267 INFO:teuthology.orchestra.run.smithi073.stdout:ID_LIKE="fedora" 2021-02-25T20:53:49.267 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION_ID="8.3" 2021-02-25T20:53:49.268 INFO:teuthology.orchestra.run.smithi073.stdout:PLATFORM_ID="platform:el8" 2021-02-25T20:53:49.268 INFO:teuthology.orchestra.run.smithi073.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-02-25T20:53:49.268 INFO:teuthology.orchestra.run.smithi073.stdout:ANSI_COLOR="0;31" 2021-02-25T20:53:49.269 INFO:teuthology.orchestra.run.smithi073.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-02-25T20:53:49.269 INFO:teuthology.orchestra.run.smithi073.stdout:HOME_URL="https://www.redhat.com/" 2021-02-25T20:53:49.269 INFO:teuthology.orchestra.run.smithi073.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-02-25T20:53:49.270 INFO:teuthology.orchestra.run.smithi073.stdout: 2021-02-25T20:53:49.270 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-02-25T20:53:49.270 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-02-25T20:53:49.271 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-02-25T20:53:49.271 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-02-25T20:53:49.272 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server 2021-02-25T20:53:49.358 INFO:teuthology.dispatcher.supervisor:Running job 5902182 2021-02-25T20:53:49.358 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/virtualenv/bin/teuthology -v --owner scheduled_xiubli@teuthology --archive /home/teuthworker/archive/xiubli-2021-02-22_04:56:06-fs:functional-wip-lxb-testing-20210221-1527-distro-basic-smithi/5902182 --name xiubli-2021-02-22_04:56:06-fs:functional-wip-lxb-testing-20210221-1527-distro-basic-smithi --description fs:functional/{begin.yaml clusters/1a3s-mds-4c-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} distro/{rhel_8.yaml} mount/fuse.yaml objectstore/bluestore-bitmap.yaml overrides/{frag_enable.yaml no_client_pidfile.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/workunit/quota.yaml} -- /home/teuthworker/archive/xiubli-2021-02-22_04:56:06-fs:functional-wip-lxb-testing-20210221-1527-distro-basic-smithi/5902182/orig.config.yaml 2021-02-25T20:53:49.365 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/xiubli-2021-02-22_04:56:06-fs:functional-wip-lxb-testing-20210221-1527-distro-basic-smithi/5902182 2021-02-25T20:53:49.366 INFO:teuthology.dispatcher.supervisor:Job PID: 10337 2021-02-25T20:53:49.367 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-02-26T08:54:08.430 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-02-26T08:54:09.348 INFO:teuthology.kill:Killing Pids: {10337} 2021-02-26T08:54:11.453 INFO:teuthology.kill:Nuking machines: ['smithi039', 'smithi073'] 2021-02-26T08:54:12.776 INFO:teuthology.kill:2021-02-26 08:54:12,776.776 INFO:teuthology.nuke:targets: 2021-02-26T08:54:12.777 INFO:teuthology.kill: smithi039.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCll/HZh+KNVdRRBLqX6igULgRmSgMMaXkWzXD+eNwurOzlUUa/e1wD1yJsareKYtKDuV+t5RTjh3oO1B1tKVnzsI4wMz/AY3IyQlKkA+JHCKSCn2hAiFGxL6dIGI96+sEUGqRqF+Fhdp1zalw+fzipEaCYrDF/OK1RYqYvJg3RcXURLm7RJ8uWsMxpBsCTjSCXrGovnPizWaZM8/XciBCmnweXEQjF7a0qbzC4NLvAqKbuzxWb5z7XRmS3/F9D4loIs0r6c6prjeH4oS5g+cNAYIOI7+CGNrnnCvD2vZNq/xCJkWSPb9vBowIfzipkAHsxzxf2q03Pl9MiHWle8Eb53L6tb+2CB5kWSBif4vaE5IQOPYg2vApG0F67nqEf5On2w9QBFXt3jadGU+V/F+d34/9+c+9W3jheT9JwE0oNTSHGEZ7PADccB+6JgsT1eFWOk96C3tJaUtnP2+CqX4JYvbUvV6+Ys2CWR81E05L+XBiEIQLn/LkoI+meYQ2M6f8= 2021-02-26T08:54:12.777 INFO:teuthology.kill: smithi073.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDITR4+xiJVX2ycb9oJtojjAbuEA58EXzk5EH0tLe3DNbmPklDW6Amm6C9aCcsOcuRGg3o1H4EQUczaxEWvuzvC+h24HpEuiZDMWwqX2rwk94W0/DHU5AUS0E7udPHtRyvgG6d3Am5iCKlrSHqiRs5lvpNZDjvsoP5XLTuDIm8eV4QoFlMXpygaLQ0L99XQbssMxwbv2PvIS59N26WRJRwoK3wjk+1061dGuuB8zYkkWtZNqPlUc0OUHnZKqYr9zUiEnFg97C6iE5mMiaNdmuUfGKCj7ifyxZYSjAUiGmUBts0l4xdIgow3bYzbxOvhoOnddQ2jx6PyxoCjHwsI872zrTKh9tTLcz56isLiQh1G1s2JrG7SlzGi6t5MqOAFdR9m+GCrrnyWx95NiECX4Gp3F/4YxPHy2qscuhMjZ0ls+Ka1DQN7RhPGurxkSwKL2Tf/CVc85pRBN6iimJKGbhql+KtQPQhcCr5UCpMNDVCQx0Dj6ofjjdq8SrAfUQBqKns= 2021-02-26T08:54:12.778 INFO:teuthology.kill:2021-02-26 08:54:12,776.776 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-26T08:54:12.785 INFO:teuthology.kill:2021-02-26 08:54:12,785.785 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-26T08:54:12.863 INFO:teuthology.kill:2021-02-26 08:54:12,863.863 INFO:teuthology.orchestra.console:Power off smithi073 2021-02-26T08:54:12.882 INFO:teuthology.kill:2021-02-26 08:54:12,882.882 INFO:teuthology.orchestra.console:Power off smithi039 2021-02-26T08:54:21.323 INFO:teuthology.kill:2021-02-26 08:54:21,323.323 INFO:teuthology.orchestra.console:Power off for smithi039 completed 2021-02-26T08:54:25.410 INFO:teuthology.kill:2021-02-26 08:54:25,410.410 INFO:teuthology.orchestra.console:Power off for smithi073 completed 2021-02-26T08:54:25.725 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2021-02-26T08:54:25.729 INFO:teuthology.task.internal:roles: ubuntu@smithi073.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2021-02-26T08:54:25.732 INFO:teuthology.misc:Compressing logs... 2021-02-26T08:54:25.745 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-26T08:54:44.176 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-02-26T08:54:44.317 INFO:teuthology.kill:No teuthology processes running 2021-02-26T08:54:46.345 INFO:teuthology.kill:Nuking machines: ['smithi039', 'smithi073'] 2021-02-26T08:54:47.914 INFO:teuthology.kill:2021-02-26 08:54:47,914.914 INFO:teuthology.nuke:targets: 2021-02-26T08:54:47.918 INFO:teuthology.kill: smithi039.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCll/HZh+KNVdRRBLqX6igULgRmSgMMaXkWzXD+eNwurOzlUUa/e1wD1yJsareKYtKDuV+t5RTjh3oO1B1tKVnzsI4wMz/AY3IyQlKkA+JHCKSCn2hAiFGxL6dIGI96+sEUGqRqF+Fhdp1zalw+fzipEaCYrDF/OK1RYqYvJg3RcXURLm7RJ8uWsMxpBsCTjSCXrGovnPizWaZM8/XciBCmnweXEQjF7a0qbzC4NLvAqKbuzxWb5z7XRmS3/F9D4loIs0r6c6prjeH4oS5g+cNAYIOI7+CGNrnnCvD2vZNq/xCJkWSPb9vBowIfzipkAHsxzxf2q03Pl9MiHWle8Eb53L6tb+2CB5kWSBif4vaE5IQOPYg2vApG0F67nqEf5On2w9QBFXt3jadGU+V/F+d34/9+c+9W3jheT9JwE0oNTSHGEZ7PADccB+6JgsT1eFWOk96C3tJaUtnP2+CqX4JYvbUvV6+Ys2CWR81E05L+XBiEIQLn/LkoI+meYQ2M6f8= 2021-02-26T08:54:47.918 INFO:teuthology.kill: smithi073.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDITR4+xiJVX2ycb9oJtojjAbuEA58EXzk5EH0tLe3DNbmPklDW6Amm6C9aCcsOcuRGg3o1H4EQUczaxEWvuzvC+h24HpEuiZDMWwqX2rwk94W0/DHU5AUS0E7udPHtRyvgG6d3Am5iCKlrSHqiRs5lvpNZDjvsoP5XLTuDIm8eV4QoFlMXpygaLQ0L99XQbssMxwbv2PvIS59N26WRJRwoK3wjk+1061dGuuB8zYkkWtZNqPlUc0OUHnZKqYr9zUiEnFg97C6iE5mMiaNdmuUfGKCj7ifyxZYSjAUiGmUBts0l4xdIgow3bYzbxOvhoOnddQ2jx6PyxoCjHwsI872zrTKh9tTLcz56isLiQh1G1s2JrG7SlzGi6t5MqOAFdR9m+GCrrnyWx95NiECX4Gp3F/4YxPHy2qscuhMjZ0ls+Ka1DQN7RhPGurxkSwKL2Tf/CVc85pRBN6iimJKGbhql+KtQPQhcCr5UCpMNDVCQx0Dj6ofjjdq8SrAfUQBqKns= 2021-02-26T08:54:47.945 INFO:teuthology.kill:2021-02-26 08:54:47,945.945 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-26T08:54:47.949 INFO:teuthology.kill:2021-02-26 08:54:47,949.949 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-26T08:54:48.027 INFO:teuthology.kill:2021-02-26 08:54:48,026.026 INFO:teuthology.orchestra.console:Power off smithi073 2021-02-26T08:54:48.052 INFO:teuthology.kill:2021-02-26 08:54:48,051.051 INFO:teuthology.orchestra.console:Power off smithi039 2021-02-26T08:54:48.211 INFO:teuthology.kill:2021-02-26 08:54:48,211.211 INFO:teuthology.orchestra.console:Power off for smithi039 completed 2021-02-26T08:54:48.217 INFO:teuthology.kill:2021-02-26 08:54:48,217.217 INFO:teuthology.orchestra.console:Power off for smithi073 completed 2021-02-26T08:54:48.355 INFO:teuthology.kill:2021-02-26 08:54:48,354.354 INFO:teuthology.lock.ops:unlocked smithi073.front.sepia.ceph.com 2021-02-26T08:54:48.358 INFO:teuthology.kill:2021-02-26 08:54:48,358.358 INFO:teuthology.lock.ops:unlocked smithi039.front.sepia.ceph.com 2021-02-26T08:56:48.583 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-02-26T08:56:48.712 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-02-26T08:56:48.715 INFO:teuthology.nuke:Checking targets against current locks 2021-02-26T08:56:48.961 INFO:teuthology.nuke:Not nuking smithi039.front.sepia.ceph.com because description doesn't match 2021-02-26T08:56:48.961 INFO:teuthology.nuke:Not nuking smithi073.front.sepia.ceph.com because description doesn't match