2021-03-01T15:20:54.905 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-01T15:20:55.019 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi074.front.sepia.ceph.com' reimaging 2021-03-01T15:20:55.021 INFO:teuthology.lock.ops:Updating [ubuntu@smithi074.front.sepia.ceph.com]: reset os type and version on server 2021-03-01T15:20:55.021 INFO:teuthology.lock.ops:Updating smithi074.front.sepia.ceph.com on lock server 2021-03-01T15:20:55.037 INFO:teuthology.lock.ops:Node 'ubuntu@smithi074.front.sepia.ceph.com' reimaging is complete 2021-03-01T15:20:55.038 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi116.front.sepia.ceph.com' reimaging 2021-03-01T15:20:55.038 INFO:teuthology.lock.ops:Updating [ubuntu@smithi116.front.sepia.ceph.com]: reset os type and version on server 2021-03-01T15:20:55.039 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2021-03-01T15:20:55.054 INFO:teuthology.lock.ops:Node 'ubuntu@smithi116.front.sepia.ceph.com' reimaging is complete 2021-03-01T15:20:55.277 INFO:teuthology.provision.fog.smithi074:Scheduling deploy of rhel 8.3 2021-03-01T15:20:55.293 INFO:teuthology.provision.fog.smithi116:Scheduling deploy of rhel 8.3 2021-03-01T15:20:55.610 INFO:teuthology.orchestra.console:Power off smithi074 2021-03-01T15:20:55.645 INFO:teuthology.orchestra.console:Power off smithi116 2021-03-01T15:21:03.978 INFO:teuthology.orchestra.console:Power off for smithi074 completed 2021-03-01T15:21:04.079 INFO:teuthology.orchestra.console:Power on smithi074 2021-03-01T15:21:08.118 INFO:teuthology.orchestra.console:Power off for smithi116 completed 2021-03-01T15:21:08.220 INFO:teuthology.orchestra.console:Power on smithi116 2021-03-01T15:21:12.444 INFO:teuthology.orchestra.console:Power on for smithi074 completed 2021-03-01T15:21:12.476 INFO:teuthology.orchestra.console:Power on for smithi116 completed 2021-03-01T15:21:12.546 INFO:teuthology.provision.fog.smithi074:Waiting for deploy to finish 2021-03-01T15:21:12.578 INFO:teuthology.provision.fog.smithi116:Waiting for deploy to finish 2021-03-01T15:25:44.478 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-01T15:25:44.775 INFO:teuthology.provision.fog.smithi116:Node is ready 2021-03-01T15:25:44.789 INFO:teuthology.orchestra.run.smithi116.stdout:smithi116.front.sepia.ceph.com 2021-03-01T15:25:44.844 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116 smithi116.front.sepia.ceph.com smithi116 2021-03-01T15:25:45.165 INFO:teuthology.provision.fog.smithi116:Deploy complete! 2021-03-01T15:25:59.679 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-01T15:25:59.974 INFO:teuthology.provision.fog.smithi074:Node is ready 2021-03-01T15:25:59.988 INFO:teuthology.orchestra.run.smithi074.stdout:smithi074.front.sepia.ceph.com 2021-03-01T15:26:00.044 INFO:teuthology.orchestra.run.smithi074.stdout:172.21.15.74 smithi074.front.sepia.ceph.com smithi074 2021-03-01T15:26:00.374 INFO:teuthology.provision.fog.smithi074:Deploy complete! 2021-03-01T15:26:00.499 INFO:teuthology.lock.ops:Checking smithi074.front.sepia.ceph.com 2021-03-01T15:26:00.500 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-01T15:26:00.519 INFO:teuthology.lock.ops:Checking smithi116.front.sepia.ceph.com 2021-03-01T15:26:00.520 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-01T15:26:00.534 INFO:teuthology.lock.ops:Updating [smithi074.front.sepia.ceph.com]: set os type and version on server 2021-03-01T15:26:00.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-01T15:26:00.815 INFO:teuthology.orchestra.run.smithi074.stdout:x86_64 2021-03-01T15:26:00.869 INFO:teuthology.orchestra.run.smithi074.stdout:NAME="Red Hat Enterprise Linux" 2021-03-01T15:26:00.869 INFO:teuthology.orchestra.run.smithi074.stdout:VERSION="8.3 (Ootpa)" 2021-03-01T15:26:00.869 INFO:teuthology.orchestra.run.smithi074.stdout:ID="rhel" 2021-03-01T15:26:00.870 INFO:teuthology.orchestra.run.smithi074.stdout:ID_LIKE="fedora" 2021-03-01T15:26:00.870 INFO:teuthology.orchestra.run.smithi074.stdout:VERSION_ID="8.3" 2021-03-01T15:26:00.870 INFO:teuthology.orchestra.run.smithi074.stdout:PLATFORM_ID="platform:el8" 2021-03-01T15:26:00.870 INFO:teuthology.orchestra.run.smithi074.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-03-01T15:26:00.870 INFO:teuthology.orchestra.run.smithi074.stdout:ANSI_COLOR="0;31" 2021-03-01T15:26:00.871 INFO:teuthology.orchestra.run.smithi074.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-03-01T15:26:00.871 INFO:teuthology.orchestra.run.smithi074.stdout:HOME_URL="https://www.redhat.com/" 2021-03-01T15:26:00.871 INFO:teuthology.orchestra.run.smithi074.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-03-01T15:26:00.872 INFO:teuthology.orchestra.run.smithi074.stdout: 2021-03-01T15:26:00.872 INFO:teuthology.orchestra.run.smithi074.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-03-01T15:26:00.872 INFO:teuthology.orchestra.run.smithi074.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-03-01T15:26:00.872 INFO:teuthology.orchestra.run.smithi074.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-03-01T15:26:00.873 INFO:teuthology.orchestra.run.smithi074.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-03-01T15:26:00.874 INFO:teuthology.lock.ops:Updating smithi074.front.sepia.ceph.com on lock server 2021-03-01T15:26:00.889 INFO:teuthology.lock.ops:Updating [smithi116.front.sepia.ceph.com]: set os type and version on server 2021-03-01T15:26:00.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-01T15:26:01.180 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2021-03-01T15:26:01.233 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="Red Hat Enterprise Linux" 2021-03-01T15:26:01.234 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="8.3 (Ootpa)" 2021-03-01T15:26:01.234 INFO:teuthology.orchestra.run.smithi116.stdout:ID="rhel" 2021-03-01T15:26:01.234 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="fedora" 2021-03-01T15:26:01.235 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="8.3" 2021-03-01T15:26:01.235 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el8" 2021-03-01T15:26:01.235 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-03-01T15:26:01.235 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2021-03-01T15:26:01.235 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-03-01T15:26:01.236 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://www.redhat.com/" 2021-03-01T15:26:01.236 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-03-01T15:26:01.236 INFO:teuthology.orchestra.run.smithi116.stdout: 2021-03-01T15:26:01.236 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-03-01T15:26:01.236 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-03-01T15:26:01.237 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-03-01T15:26:01.237 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-03-01T15:26:01.237 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2021-03-01T15:26:01.373 INFO:teuthology.dispatcher.supervisor:Running job 5923500 2021-03-01T15:26:01.374 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology -v --owner scheduled_xiubli@teuthology --archive /home/teuthworker/archive/xiubli-2021-03-01_12:38:27-fs:functional-wip-testing-20210301-1545-distro-basic-smithi/5923500 --name xiubli-2021-03-01_12:38:27-fs:functional-wip-testing-20210301-1545-distro-basic-smithi --description fs:functional/{begin.yaml clusters/1a3s-mds-4c-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} distro/{rhel_8.yaml} mount/fuse.yaml objectstore/bluestore-bitmap.yaml overrides/{frag_enable.yaml no_client_pidfile.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/snapshots.yaml} -- /home/teuthworker/archive/xiubli-2021-03-01_12:38:27-fs:functional-wip-testing-20210301-1545-distro-basic-smithi/5923500/orig.config.yaml 2021-03-01T15:26:01.377 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/xiubli-2021-03-01_12:38:27-fs:functional-wip-testing-20210301-1545-distro-basic-smithi/5923500 2021-03-01T15:26:01.378 INFO:teuthology.dispatcher.supervisor:Job PID: 10333 2021-03-01T15:26:01.379 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-03-02T03:26:52.275 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-02T03:26:53.680 INFO:teuthology.kill:Killing Pids: {10333} 2021-03-02T03:26:56.157 INFO:teuthology.kill:Nuking machines: ['smithi074', 'smithi116'] 2021-03-02T03:26:57.675 INFO:teuthology.kill:2021-03-02 03:26:57,675.675 INFO:teuthology.nuke:targets: 2021-03-02T03:26:57.676 INFO:teuthology.kill: smithi074.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxuflwWDMVWOnRjiK2JQZSByXteQGplS/7QwTMzYd/YKb1m+K54WgwhPR7Ox6XEo4qhnhf34M7bW69cj19E3NefHtJi3GZM9SrkMBj4zLzYAHHmuqmAsvLoDmkEpJhlJZuW26nCsp1jjiZUKhMFFmVHBeZwM6tNoGHXYBIgL5v3giMh6sVYU2Kg2R0r+vfdF4fc26QH4+NKtwLvuTPXG8w36XA1MTIRskeQoSPvECx2lb3wCANDhFl35kTqAK7hzhELj27yo+p7oyRRPBQqIlTkIxYCiTpeZd8mtY45SgD9fn+ge8UuZZZ+Ej0C83hrmzIFe1gww5RWgORwrCOJy3GHG3YXfOA+WAUr0dD7rj4pW+HlgGqKWCJCCOJ7LFN6iGcv5Z7IuWwe/hH/9VQuyW4YdlQp7lAJXnRvu7ZpQROB2048R83pVJ4uFat8ddkb2V5a23ocOZN+7MzRskoOnvEQYEzEyMk9LomlAh7qo8u01ODxyBO+phUUzncybaAtuc= 2021-03-02T03:26:57.677 INFO:teuthology.kill: smithi116.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE9mkOUbsoDb+OWNnKnP9yJ+TXa1scvl+myHtAydaPP56n7ir5x226jOPQBQKQvqcR7zM4MjWSaAgPjjIFOuH2+DcCm15/37c4CjbtIb7tfTJ6k/EuNfwy2i224DMf7TbglYlIOkyI2ePQcTGJIaHOXSTBWOq8U1LpZqPH1bm8fwXMQlYmDVw9K24XRZXPlGRboiuqcsn2xiildmvKBaoM5qBZPQtWexjR8b3E+8d8OoALmoYctmF9212OIxksH6ocsOtEZgLlSs1ImTY9dyBN8yp7IrbneCY512HFN8jfKN1QVzyvX7FuSenfIcBQp9CLnyLqcrVFqjkgeXc/7XHravdUD1BUC0aUGqUlIOPXpzP055DlyknnHGEIEeNzfheG25hvt1KlHdbf8R/40tLVSfp8d299Bq0Lv02kMGDK8WNa5DT7FnZgIsHruMVfssMuSFWl5D1RgyBlkfjkS822hNemNt22JisySB0CrbcknVe4jo/t3OtZwUI4O+fpsxk= 2021-03-02T03:26:57.677 INFO:teuthology.kill:2021-03-02 03:26:57,677.677 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-02T03:26:57.680 INFO:teuthology.kill:2021-03-02 03:26:57,680.680 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-02T03:26:57.746 INFO:teuthology.kill:2021-03-02 03:26:57,746.746 INFO:teuthology.orchestra.console:Power off smithi116 2021-03-02T03:26:57.763 INFO:teuthology.kill:2021-03-02 03:26:57,763.763 INFO:teuthology.orchestra.console:Power off smithi074 2021-03-02T03:27:06.135 INFO:teuthology.kill:2021-03-02 03:27:06,134.134 INFO:teuthology.orchestra.console:Power off for smithi074 completed 2021-03-02T03:27:10.236 INFO:teuthology.kill:2021-03-02 03:27:10,235.235 INFO:teuthology.orchestra.console:Power off for smithi116 completed 2021-03-02T03:27:10.511 INFO:teuthology.task.internal:roles: ubuntu@smithi074.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2021-03-02T03:27:10.511 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2021-03-02T03:27:10.512 INFO:teuthology.misc:Compressing logs... 2021-03-02T03:27:10.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-02T03:28:10.575 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-02T03:28:10.643 INFO:teuthology.kill:No teuthology processes running 2021-03-02T03:28:11.730 INFO:teuthology.kill:Nuking machines: ['smithi074', 'smithi116'] 2021-03-02T03:28:12.703 INFO:teuthology.kill:2021-03-02 03:28:12,703.703 INFO:teuthology.nuke:targets: 2021-03-02T03:28:12.704 INFO:teuthology.kill: smithi074.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxuflwWDMVWOnRjiK2JQZSByXteQGplS/7QwTMzYd/YKb1m+K54WgwhPR7Ox6XEo4qhnhf34M7bW69cj19E3NefHtJi3GZM9SrkMBj4zLzYAHHmuqmAsvLoDmkEpJhlJZuW26nCsp1jjiZUKhMFFmVHBeZwM6tNoGHXYBIgL5v3giMh6sVYU2Kg2R0r+vfdF4fc26QH4+NKtwLvuTPXG8w36XA1MTIRskeQoSPvECx2lb3wCANDhFl35kTqAK7hzhELj27yo+p7oyRRPBQqIlTkIxYCiTpeZd8mtY45SgD9fn+ge8UuZZZ+Ej0C83hrmzIFe1gww5RWgORwrCOJy3GHG3YXfOA+WAUr0dD7rj4pW+HlgGqKWCJCCOJ7LFN6iGcv5Z7IuWwe/hH/9VQuyW4YdlQp7lAJXnRvu7ZpQROB2048R83pVJ4uFat8ddkb2V5a23ocOZN+7MzRskoOnvEQYEzEyMk9LomlAh7qo8u01ODxyBO+phUUzncybaAtuc= 2021-03-02T03:28:12.705 INFO:teuthology.kill: smithi116.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE9mkOUbsoDb+OWNnKnP9yJ+TXa1scvl+myHtAydaPP56n7ir5x226jOPQBQKQvqcR7zM4MjWSaAgPjjIFOuH2+DcCm15/37c4CjbtIb7tfTJ6k/EuNfwy2i224DMf7TbglYlIOkyI2ePQcTGJIaHOXSTBWOq8U1LpZqPH1bm8fwXMQlYmDVw9K24XRZXPlGRboiuqcsn2xiildmvKBaoM5qBZPQtWexjR8b3E+8d8OoALmoYctmF9212OIxksH6ocsOtEZgLlSs1ImTY9dyBN8yp7IrbneCY512HFN8jfKN1QVzyvX7FuSenfIcBQp9CLnyLqcrVFqjkgeXc/7XHravdUD1BUC0aUGqUlIOPXpzP055DlyknnHGEIEeNzfheG25hvt1KlHdbf8R/40tLVSfp8d299Bq0Lv02kMGDK8WNa5DT7FnZgIsHruMVfssMuSFWl5D1RgyBlkfjkS822hNemNt22JisySB0CrbcknVe4jo/t3OtZwUI4O+fpsxk= 2021-03-02T03:28:12.727 INFO:teuthology.kill:2021-03-02 03:28:12,727.727 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-02T03:28:12.730 INFO:teuthology.kill:2021-03-02 03:28:12,729.729 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-02T03:28:12.795 INFO:teuthology.kill:2021-03-02 03:28:12,795.795 INFO:teuthology.orchestra.console:Power off smithi116 2021-03-02T03:28:12.814 INFO:teuthology.kill:2021-03-02 03:28:12,813.813 INFO:teuthology.orchestra.console:Power off smithi074 2021-03-02T03:28:12.946 INFO:teuthology.kill:2021-03-02 03:28:12,946.946 INFO:teuthology.orchestra.console:Power off for smithi116 completed 2021-03-02T03:28:12.958 INFO:teuthology.kill:2021-03-02 03:28:12,957.957 INFO:teuthology.orchestra.console:Power off for smithi074 completed 2021-03-02T03:28:13.093 INFO:teuthology.kill:2021-03-02 03:28:13,093.093 INFO:teuthology.lock.ops:unlocked smithi116.front.sepia.ceph.com 2021-03-02T03:28:13.098 INFO:teuthology.kill:2021-03-02 03:28:13,098.098 INFO:teuthology.lock.ops:unlocked smithi074.front.sepia.ceph.com 2021-03-02T03:30:13.399 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-03-02T03:30:13.503 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-03-02T03:30:13.504 INFO:teuthology.nuke:Checking targets against current locks 2021-03-02T03:30:13.584 INFO:teuthology.nuke:Not nuking smithi074.front.sepia.ceph.com because description doesn't match 2021-03-02T03:30:13.585 INFO:teuthology.nuke:Not nuking smithi116.front.sepia.ceph.com because description doesn't match