2021-05-28T05:25:39.346 INFO:root:teuthology version: 1.1.0-f359b10d 2021-05-28T05:25:39.470 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi089.front.sepia.ceph.com' reimaging 2021-05-28T05:25:39.471 INFO:teuthology.lock.ops:Updating [ubuntu@smithi089.front.sepia.ceph.com]: reset os type and version on server 2021-05-28T05:25:39.472 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2021-05-28T05:25:39.505 INFO:teuthology.lock.ops:Node 'ubuntu@smithi089.front.sepia.ceph.com' reimaging is complete 2021-05-28T05:25:39.505 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi142.front.sepia.ceph.com' reimaging 2021-05-28T05:25:39.506 INFO:teuthology.lock.ops:Updating [ubuntu@smithi142.front.sepia.ceph.com]: reset os type and version on server 2021-05-28T05:25:39.506 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2021-05-28T05:25:39.521 INFO:teuthology.lock.ops:Node 'ubuntu@smithi142.front.sepia.ceph.com' reimaging is complete 2021-05-28T05:25:39.708 INFO:teuthology.provision.fog.smithi089:Scheduling deploy of centos 8.3 2021-05-28T05:25:39.728 INFO:teuthology.provision.fog.smithi142:Scheduling deploy of centos 8.3 2021-05-28T05:25:40.072 INFO:teuthology.orchestra.console:Power off smithi089 2021-05-28T05:25:40.143 INFO:teuthology.orchestra.console:Power off smithi142 2021-05-28T05:25:48.531 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2021-05-28T05:25:48.633 INFO:teuthology.orchestra.console:Power on smithi142 2021-05-28T05:25:52.602 INFO:teuthology.orchestra.console:Power off for smithi089 completed 2021-05-28T05:25:52.704 INFO:teuthology.orchestra.console:Power on smithi089 2021-05-28T05:25:56.971 INFO:teuthology.orchestra.console:Power on for smithi089 completed 2021-05-28T05:25:57.021 INFO:teuthology.orchestra.console:Power on for smithi142 completed 2021-05-28T05:25:57.074 INFO:teuthology.provision.fog.smithi089:Waiting for deploy to finish 2021-05-28T05:25:57.123 INFO:teuthology.provision.fog.smithi142:Waiting for deploy to finish 2021-05-28T05:28:28.418 ERROR:teuthology.orchestra.connection:Error authenticating with smithi089.front.sepia.ceph.com: Authentication failed. 2021-05-28T05:30:04.477 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-28T05:30:25.215 INFO:teuthology.provision.fog.smithi089:Node is ready 2021-05-28T05:30:25.235 INFO:teuthology.orchestra.run.smithi089.stdout:smithi089.front.sepia.ceph.com 2021-05-28T05:30:25.291 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89 smithi089.front.sepia.ceph.com smithi089 2021-05-28T05:30:25.640 INFO:teuthology.provision.fog.smithi089:Deploy complete! 2021-05-28T05:30:29.000 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-28T05:30:29.316 INFO:teuthology.provision.fog.smithi142:Node is ready 2021-05-28T05:30:29.330 INFO:teuthology.orchestra.run.smithi142.stdout:smithi142.front.sepia.ceph.com 2021-05-28T05:30:29.386 INFO:teuthology.orchestra.run.smithi142.stdout:172.21.15.142 smithi142.front.sepia.ceph.com smithi142 2021-05-28T05:30:29.700 INFO:teuthology.provision.fog.smithi142:Deploy complete! 2021-05-28T05:30:29.816 INFO:teuthology.lock.ops:Checking smithi089.front.sepia.ceph.com 2021-05-28T05:30:29.817 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-28T05:30:29.833 INFO:teuthology.lock.ops:Checking smithi142.front.sepia.ceph.com 2021-05-28T05:30:29.833 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-28T05:30:29.847 INFO:teuthology.lock.ops:Updating [smithi089.front.sepia.ceph.com]: set os type and version on server 2021-05-28T05:30:29.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-28T05:30:30.132 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2021-05-28T05:30:30.186 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="CentOS Linux" 2021-05-28T05:30:30.187 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="8" 2021-05-28T05:30:30.187 INFO:teuthology.orchestra.run.smithi089.stdout:ID="centos" 2021-05-28T05:30:30.187 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE="rhel fedora" 2021-05-28T05:30:30.188 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="8" 2021-05-28T05:30:30.188 INFO:teuthology.orchestra.run.smithi089.stdout:PLATFORM_ID="platform:el8" 2021-05-28T05:30:30.188 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="CentOS Linux 8" 2021-05-28T05:30:30.188 INFO:teuthology.orchestra.run.smithi089.stdout:ANSI_COLOR="0;31" 2021-05-28T05:30:30.188 INFO:teuthology.orchestra.run.smithi089.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-28T05:30:30.189 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://centos.org/" 2021-05-28T05:30:30.189 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-28T05:30:30.189 INFO:teuthology.orchestra.run.smithi089.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-28T05:30:30.189 INFO:teuthology.orchestra.run.smithi089.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-28T05:30:30.190 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2021-05-28T05:30:30.205 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: set os type and version on server 2021-05-28T05:30:30.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-28T05:30:30.420 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2021-05-28T05:30:30.474 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="CentOS Linux" 2021-05-28T05:30:30.475 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="8" 2021-05-28T05:30:30.475 INFO:teuthology.orchestra.run.smithi142.stdout:ID="centos" 2021-05-28T05:30:30.475 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="rhel fedora" 2021-05-28T05:30:30.475 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="8" 2021-05-28T05:30:30.476 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el8" 2021-05-28T05:30:30.476 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="CentOS Linux 8" 2021-05-28T05:30:30.476 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31" 2021-05-28T05:30:30.476 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-28T05:30:30.476 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://centos.org/" 2021-05-28T05:30:30.477 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-28T05:30:30.477 INFO:teuthology.orchestra.run.smithi142.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-28T05:30:30.477 INFO:teuthology.orchestra.run.smithi142.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-28T05:30:30.478 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2021-05-28T05:30:30.548 INFO:teuthology.dispatcher.supervisor:Running job 6140297 2021-05-28T05:30:30.548 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology -v --owner scheduled_xiubli@teuthology --archive /home/teuthworker/archive/xiubli-2021-05-28_05:23:05-fs:functional:-wip-lxb-20210527-45434-pg-num3-distro-basic-smithi/6140297 --name xiubli-2021-05-28_05:23:05-fs:functional:-wip-lxb-20210527-45434-pg-num3-distro-basic-smithi --description fs:functional:/{begin.yaml clusters/1a3s-mds-4c-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} distro/{centos_8.yaml} mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore/bluestore-ec-root.yaml overrides/{frag_enable.yaml no_client_pidfile.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/mds-full.yaml} -- /home/teuthworker/archive/xiubli-2021-05-28_05:23:05-fs:functional:-wip-lxb-20210527-45434-pg-num3-distro-basic-smithi/6140297/orig.config.yaml 2021-05-28T05:30:30.552 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/xiubli-2021-05-28_05:23:05-fs:functional:-wip-lxb-20210527-45434-pg-num3-distro-basic-smithi/6140297 2021-05-28T05:30:30.553 INFO:teuthology.dispatcher.supervisor:Job PID: 15917 2021-05-28T05:30:30.553 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-28T17:30:45.466 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-28T17:30:45.648 INFO:teuthology.kill:Killing Pids: {15917} 2021-05-28T17:30:46.595 INFO:teuthology.kill:Nuking machines: ['smithi089', 'smithi142'] 2021-05-28T17:30:47.383 INFO:teuthology.kill:2021-05-28 17:30:47,383.383 INFO:teuthology.nuke:targets: 2021-05-28T17:30:47.384 INFO:teuthology.kill: smithi089.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCo8U2AO3Kd13Av6x0sWq7S5N3/MrbHPXrcYHGT4ftFYK3ex8L6ySn/pg7kXEtOzZWi9AqbTFqiEpKEAF8guEZX0eLJxTT9YieC95/1bOWsyFwTFYfcDKsJOckxVBKelIqOnVs/TBofTzdByMtLuExLgpNN7eBQ4QeQ9vROA8PlEcaCz2Ust43fMBtGJV9piBU8+3SVpkl2KfKZb+dFlDPiXxDT3xC5Gqvr4R3am/oRyImWiynEzB6uY6VHNz5cWbcMRX+diFFWtS/+5b9lp5FQbK/gQVWVzbb+At71NfIgeXdsyoIFHJtAHFCE/3R2VBsSlXI+P/nFsvUgsAouqKzVt4puCfrqaPw1irVixEpDPb7kwXlM1L9Js/z0GK7MFCnAMysEqckrHeomkfUVMsoF7GezQmctT+DRiwbRRd9Sx6cWAVM1BFgfNjaqSZeSStnXTyvjx5GkD+vS8ROMP7UNlzGxB53R6tYMGwIWGjIcpRkZpjrx+8SZJqKagME8uZE= 2021-05-28T17:30:47.384 INFO:teuthology.kill: smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRNuX8LSaYFOP7081tz+jfZbWenE8xElmUZ2qdBSmyi5ANCZ+OFe5yj+p2vudHwcauViaG62pevDajNdXVXFXDIoN0S14vIriu3vhOTo3K91Iirjnhtb2LRDFMqLSOVCvWJHfWw+bmUeETwjawzNuzjVmPGboOkxdTtjFPNjcOgsgfzaP0YWq0BHc5VR7n8QqyeT/bU/gECMtaYookYGG1kpd2mvNw3ufqhqLTf7ob7lhsmshVZDMYfyopzsOdNkiTYaXJSQDyDaM1qG4WKhziRVhR1hKvsVULExMeUL2pZDAloeknZfsohh1sEufx2GH1HWSfvLwrEcvV1ZzKuh7szChXQAhBA0BVFLQfmC0jP9QrXt8onaISAcoT12AxUZcGec4RuRn8FBx1kHnIkpUNagNYMjnMJqfjkY5AMzivlp0pjHbZVv+33/tg4Hh3lTwqVy4ra6RDpbwhfegXBLxrV+UrY0QElea3aFuyOX6W0yBTUliub542UH908GtzcgU= 2021-05-28T17:30:47.385 INFO:teuthology.kill:2021-05-28 17:30:47,383.383 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-28T17:30:47.385 INFO:teuthology.kill:2021-05-28 17:30:47,385.385 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-28T17:30:47.439 INFO:teuthology.kill:2021-05-28 17:30:47,439.439 INFO:teuthology.orchestra.console:Power off smithi089 2021-05-28T17:30:47.451 INFO:teuthology.kill:2021-05-28 17:30:47,451.451 INFO:teuthology.orchestra.console:Power off smithi142 2021-05-28T17:30:59.929 INFO:teuthology.kill:2021-05-28 17:30:59,929.929 INFO:teuthology.orchestra.console:Power off for smithi089 completed 2021-05-28T17:30:59.934 INFO:teuthology.kill:2021-05-28 17:30:59,934.934 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2021-05-28T17:31:00.160 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2021-05-28T17:31:00.161 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2021-05-28T17:31:00.161 INFO:teuthology.misc:Compressing logs... 2021-05-28T17:31:00.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-28T17:32:00.166 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-28T17:32:00.225 INFO:teuthology.kill:No teuthology processes running 2021-05-28T17:32:01.154 INFO:teuthology.kill:Nuking machines: ['smithi089', 'smithi142'] 2021-05-28T17:32:01.958 INFO:teuthology.kill:2021-05-28 17:32:01,958.958 INFO:teuthology.nuke:targets: 2021-05-28T17:32:01.959 INFO:teuthology.kill: smithi089.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCo8U2AO3Kd13Av6x0sWq7S5N3/MrbHPXrcYHGT4ftFYK3ex8L6ySn/pg7kXEtOzZWi9AqbTFqiEpKEAF8guEZX0eLJxTT9YieC95/1bOWsyFwTFYfcDKsJOckxVBKelIqOnVs/TBofTzdByMtLuExLgpNN7eBQ4QeQ9vROA8PlEcaCz2Ust43fMBtGJV9piBU8+3SVpkl2KfKZb+dFlDPiXxDT3xC5Gqvr4R3am/oRyImWiynEzB6uY6VHNz5cWbcMRX+diFFWtS/+5b9lp5FQbK/gQVWVzbb+At71NfIgeXdsyoIFHJtAHFCE/3R2VBsSlXI+P/nFsvUgsAouqKzVt4puCfrqaPw1irVixEpDPb7kwXlM1L9Js/z0GK7MFCnAMysEqckrHeomkfUVMsoF7GezQmctT+DRiwbRRd9Sx6cWAVM1BFgfNjaqSZeSStnXTyvjx5GkD+vS8ROMP7UNlzGxB53R6tYMGwIWGjIcpRkZpjrx+8SZJqKagME8uZE= 2021-05-28T17:32:01.959 INFO:teuthology.kill: smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRNuX8LSaYFOP7081tz+jfZbWenE8xElmUZ2qdBSmyi5ANCZ+OFe5yj+p2vudHwcauViaG62pevDajNdXVXFXDIoN0S14vIriu3vhOTo3K91Iirjnhtb2LRDFMqLSOVCvWJHfWw+bmUeETwjawzNuzjVmPGboOkxdTtjFPNjcOgsgfzaP0YWq0BHc5VR7n8QqyeT/bU/gECMtaYookYGG1kpd2mvNw3ufqhqLTf7ob7lhsmshVZDMYfyopzsOdNkiTYaXJSQDyDaM1qG4WKhziRVhR1hKvsVULExMeUL2pZDAloeknZfsohh1sEufx2GH1HWSfvLwrEcvV1ZzKuh7szChXQAhBA0BVFLQfmC0jP9QrXt8onaISAcoT12AxUZcGec4RuRn8FBx1kHnIkpUNagNYMjnMJqfjkY5AMzivlp0pjHbZVv+33/tg4Hh3lTwqVy4ra6RDpbwhfegXBLxrV+UrY0QElea3aFuyOX6W0yBTUliub542UH908GtzcgU= 2021-05-28T17:32:01.977 INFO:teuthology.kill:2021-05-28 17:32:01,977.977 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-28T17:32:01.980 INFO:teuthology.kill:2021-05-28 17:32:01,980.980 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-28T17:32:02.033 INFO:teuthology.kill:2021-05-28 17:32:02,032.032 INFO:teuthology.orchestra.console:Power off smithi089 2021-05-28T17:32:02.046 INFO:teuthology.kill:2021-05-28 17:32:02,046.046 INFO:teuthology.orchestra.console:Power off smithi142 2021-05-28T17:32:02.175 INFO:teuthology.kill:2021-05-28 17:32:02,174.174 INFO:teuthology.orchestra.console:Power off for smithi089 completed 2021-05-28T17:32:02.187 INFO:teuthology.kill:2021-05-28 17:32:02,187.187 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2021-05-28T17:32:02.337 INFO:teuthology.kill:2021-05-28 17:32:02,336.336 INFO:teuthology.lock.ops:unlocked smithi089.front.sepia.ceph.com 2021-05-28T17:32:02.355 INFO:teuthology.kill:2021-05-28 17:32:02,355.355 INFO:teuthology.lock.ops:unlocked smithi142.front.sepia.ceph.com 2021-05-28T17:34:02.513 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15