2021-05-27T11:53:06.607 INFO:root:teuthology version: 1.1.0-f359b10d 2021-05-27T11:53:06.698 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi031.front.sepia.ceph.com' reimaging 2021-05-27T11:53:06.699 INFO:teuthology.lock.ops:Updating [ubuntu@smithi031.front.sepia.ceph.com]: reset os type and version on server 2021-05-27T11:53:06.700 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2021-05-27T11:53:06.714 INFO:teuthology.lock.ops:Node 'ubuntu@smithi031.front.sepia.ceph.com' reimaging is complete 2021-05-27T11:53:06.715 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi143.front.sepia.ceph.com' reimaging 2021-05-27T11:53:06.715 INFO:teuthology.lock.ops:Updating [ubuntu@smithi143.front.sepia.ceph.com]: reset os type and version on server 2021-05-27T11:53:06.715 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2021-05-27T11:53:06.730 INFO:teuthology.lock.ops:Node 'ubuntu@smithi143.front.sepia.ceph.com' reimaging is complete 2021-05-27T11:53:06.909 INFO:teuthology.provision.fog.smithi031:Scheduling deploy of centos 8.3 2021-05-27T11:53:06.926 INFO:teuthology.provision.fog.smithi143:Scheduling deploy of centos 8.3 2021-05-27T11:53:07.205 INFO:teuthology.orchestra.console:Power off smithi031 2021-05-27T11:53:07.258 INFO:teuthology.orchestra.console:Power off smithi143 2021-05-27T11:53:07.347 INFO:teuthology.orchestra.console:Power off for smithi031 completed 2021-05-27T11:53:07.397 INFO:teuthology.orchestra.console:Power off for smithi143 completed 2021-05-27T11:53:07.449 INFO:teuthology.orchestra.console:Power on smithi031 2021-05-27T11:53:07.498 INFO:teuthology.orchestra.console:Power on smithi143 2021-05-27T11:53:11.704 INFO:teuthology.orchestra.console:Power on for smithi031 completed 2021-05-27T11:53:11.749 INFO:teuthology.orchestra.console:Power on for smithi143 completed 2021-05-27T11:53:11.806 INFO:teuthology.provision.fog.smithi031:Waiting for deploy to finish 2021-05-27T11:53:11.851 INFO:teuthology.provision.fog.smithi143:Waiting for deploy to finish 2021-05-27T11:57:25.996 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-27T11:57:26.348 INFO:teuthology.provision.fog.smithi031:Node is ready 2021-05-27T11:57:26.403 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031.front.sepia.ceph.com 2021-05-27T11:57:26.465 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31 smithi031.front.sepia.ceph.com smithi031 2021-05-27T11:57:26.795 INFO:teuthology.provision.fog.smithi031:Deploy complete! 2021-05-27T11:57:56.191 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-27T11:57:56.515 INFO:teuthology.provision.fog.smithi143:Node is ready 2021-05-27T11:57:56.529 INFO:teuthology.orchestra.run.smithi143.stdout:smithi143.front.sepia.ceph.com 2021-05-27T11:57:56.583 INFO:teuthology.orchestra.run.smithi143.stdout:172.21.15.143 smithi143.front.sepia.ceph.com smithi143 2021-05-27T11:57:56.903 INFO:teuthology.provision.fog.smithi143:Deploy complete! 2021-05-27T11:57:57.036 INFO:teuthology.lock.ops:Checking smithi031.front.sepia.ceph.com 2021-05-27T11:57:57.036 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-27T11:57:57.050 INFO:teuthology.lock.ops:Checking smithi143.front.sepia.ceph.com 2021-05-27T11:57:57.050 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-27T11:57:57.063 INFO:teuthology.lock.ops:Updating [smithi031.front.sepia.ceph.com]: set os type and version on server 2021-05-27T11:57:57.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-27T11:57:57.364 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2021-05-27T11:57:57.420 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Linux" 2021-05-27T11:57:57.420 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="8" 2021-05-27T11:57:57.420 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2021-05-27T11:57:57.420 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2021-05-27T11:57:57.421 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="8" 2021-05-27T11:57:57.421 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el8" 2021-05-27T11:57:57.421 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Linux 8" 2021-05-27T11:57:57.421 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2021-05-27T11:57:57.421 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-27T11:57:57.422 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2021-05-27T11:57:57.422 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-27T11:57:57.422 INFO:teuthology.orchestra.run.smithi031.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-27T11:57:57.422 INFO:teuthology.orchestra.run.smithi031.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-27T11:57:57.423 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2021-05-27T11:57:57.437 INFO:teuthology.lock.ops:Updating [smithi143.front.sepia.ceph.com]: set os type and version on server 2021-05-27T11:57:57.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-27T11:57:57.719 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64 2021-05-27T11:57:57.773 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="CentOS Linux" 2021-05-27T11:57:57.773 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="8" 2021-05-27T11:57:57.774 INFO:teuthology.orchestra.run.smithi143.stdout:ID="centos" 2021-05-27T11:57:57.774 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE="rhel fedora" 2021-05-27T11:57:57.774 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="8" 2021-05-27T11:57:57.774 INFO:teuthology.orchestra.run.smithi143.stdout:PLATFORM_ID="platform:el8" 2021-05-27T11:57:57.775 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="CentOS Linux 8" 2021-05-27T11:57:57.775 INFO:teuthology.orchestra.run.smithi143.stdout:ANSI_COLOR="0;31" 2021-05-27T11:57:57.775 INFO:teuthology.orchestra.run.smithi143.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-27T11:57:57.775 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://centos.org/" 2021-05-27T11:57:57.776 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-27T11:57:57.776 INFO:teuthology.orchestra.run.smithi143.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-27T11:57:57.776 INFO:teuthology.orchestra.run.smithi143.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-27T11:57:57.777 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2021-05-27T11:57:57.847 INFO:teuthology.dispatcher.supervisor:Running job 6137847 2021-05-27T11:57:57.847 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology -v --owner scheduled_xiubli@teuthology --archive /home/teuthworker/archive/xiubli-2021-05-27_11:52:54-fs:functional:-wip-lxb-20210524-45434-pg-num-distro-basic-smithi/6137847 --name xiubli-2021-05-27_11:52:54-fs:functional:-wip-lxb-20210524-45434-pg-num-distro-basic-smithi --description fs:functional:/{begin.yaml clusters/1a3s-mds-4c-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} distro/{ubuntu_latest.yaml} mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore/bluestore-ec-root.yaml overrides/{frag_enable.yaml no_client_pidfile.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/mds-full.yaml} -- /home/teuthworker/archive/xiubli-2021-05-27_11:52:54-fs:functional:-wip-lxb-20210524-45434-pg-num-distro-basic-smithi/6137847/orig.config.yaml 2021-05-27T11:57:57.850 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/xiubli-2021-05-27_11:52:54-fs:functional:-wip-lxb-20210524-45434-pg-num-distro-basic-smithi/6137847 2021-05-27T11:57:57.851 INFO:teuthology.dispatcher.supervisor:Job PID: 24030 2021-05-27T11:57:57.851 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-27T23:58:46.571 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-27T23:58:46.744 INFO:teuthology.kill:Killing Pids: {24030} 2021-05-27T23:58:47.877 INFO:teuthology.kill:Nuking machines: ['smithi031', 'smithi143'] 2021-05-27T23:58:48.877 INFO:teuthology.kill:2021-05-27 23:58:48,876.876 INFO:teuthology.nuke:targets: 2021-05-27T23:58:48.878 INFO:teuthology.kill: smithi031.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzpvS7uWvGAcbZr+cucv2OBo3mMxdgfhuK4s643sT0CKh1Eo2Smy0/7dDp/QwXeIoE3NGO6nmvElkqLoI/dh6eli9ne2fvoRTdwu23cTsgFOg9skF1PXbelM2VnN3TZ7CG1iK27jqyjns4aYg8CiYTh9Pxtkqt7Mh6KI7ch7nJwOaAj5Ow5vXBopDemX8a5MAmvycpJo/8WVzAOl4DGw8nwLRa2Fna0Ii0AKg0XoLqBj7EA7fHidCKRePDgnisp81jR9nGF0xsPZUrlJXZb/NnulpJLOTSk1iuYL87Tws2LwpGX/9nL5bDX9ffGHMljIEcbT1GEZ+ZZgxLJ7AfvYL5BnG+f3MQZxy8tkUqDQEnLkIyUtRD1ytzzDUFQYd02MhxGhIKxJHwM5LNXbbyZaJ1R+K4e1zHuKVh4n/KbCJcAwlCZ+pI0gcuDcFMOw42+iokF70FePVvFFwOnG8Mrb+YzQYcsQlWNXuCbaNp2NVf4DusaddMUQr2MlbsmjOFokM= 2021-05-27T23:58:48.878 INFO:teuthology.kill: smithi143.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAeTrNxzpcCoX7ypsYaneePSg7ist82aPHoigR9h/DFYsvlvAFlfp7XFos+qtPIBxhEsb9smRw4wOqCqZEYvtPbE0Bet0KXnVg/DqM5ADOdV4q6rkli1GJgrFExMJJh4Oi2e8dTLegqzMCJyi5j1Q8RsfoiX/QaE7vKe+8/6QqbnJ2CYK9/Bn8ECQKFamC69QkR+71TCnehtH3fv6OQ8E7pKArUoG8DF3Fl5sZ1b/z4WP/L2LidroODYlpkcj5qhI781Or9h9SQI6x587AmYTTbxCiSpxMoFAM1NsGHwqGT0sgx/ivT7/hbSaqHbFOhGAK1mHqokrAj07z0+t8Cxw7FugtXKtXocJ/95q7elrBWgkzeyLGjRIKy//OqPcJglEJhcZyPvFtDbuovsl1QnLxrbuMmHyVdaII7S4agozHIiMwunGiQWBZG9ZAxSW90b2fkzm8HTxguQISSkFLeZLj5ok/cfYnYTZU4m0dBm968bBrEO/z48gV7kIQhcD6asc= 2021-05-27T23:58:48.878 INFO:teuthology.kill:2021-05-27 23:58:48,877.877 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-27T23:58:48.881 INFO:teuthology.kill:2021-05-27 23:58:48,879.879 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-27T23:58:48.946 INFO:teuthology.kill:2021-05-27 23:58:48,945.945 INFO:teuthology.orchestra.console:Power off smithi031 2021-05-27T23:58:48.963 INFO:teuthology.kill:2021-05-27 23:58:48,962.962 INFO:teuthology.orchestra.console:Power off smithi143 2021-05-27T23:59:01.435 INFO:teuthology.kill:2021-05-27 23:59:01,435.435 INFO:teuthology.orchestra.console:Power off for smithi031 completed 2021-05-27T23:59:01.440 INFO:teuthology.kill:2021-05-27 23:59:01,440.440 INFO:teuthology.orchestra.console:Power off for smithi143 completed 2021-05-27T23:59:01.757 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2021-05-27T23:59:01.758 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2021-05-27T23:59:01.759 INFO:teuthology.misc:Compressing logs... 2021-05-27T23:59:01.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-27T23:59:20.271 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-27T23:59:20.356 INFO:teuthology.kill:No teuthology processes running 2021-05-27T23:59:21.514 INFO:teuthology.kill:Nuking machines: ['smithi031', 'smithi143'] 2021-05-27T23:59:22.540 INFO:teuthology.kill:2021-05-27 23:59:22,540.540 INFO:teuthology.nuke:targets: 2021-05-27T23:59:22.541 INFO:teuthology.kill: smithi031.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzpvS7uWvGAcbZr+cucv2OBo3mMxdgfhuK4s643sT0CKh1Eo2Smy0/7dDp/QwXeIoE3NGO6nmvElkqLoI/dh6eli9ne2fvoRTdwu23cTsgFOg9skF1PXbelM2VnN3TZ7CG1iK27jqyjns4aYg8CiYTh9Pxtkqt7Mh6KI7ch7nJwOaAj5Ow5vXBopDemX8a5MAmvycpJo/8WVzAOl4DGw8nwLRa2Fna0Ii0AKg0XoLqBj7EA7fHidCKRePDgnisp81jR9nGF0xsPZUrlJXZb/NnulpJLOTSk1iuYL87Tws2LwpGX/9nL5bDX9ffGHMljIEcbT1GEZ+ZZgxLJ7AfvYL5BnG+f3MQZxy8tkUqDQEnLkIyUtRD1ytzzDUFQYd02MhxGhIKxJHwM5LNXbbyZaJ1R+K4e1zHuKVh4n/KbCJcAwlCZ+pI0gcuDcFMOw42+iokF70FePVvFFwOnG8Mrb+YzQYcsQlWNXuCbaNp2NVf4DusaddMUQr2MlbsmjOFokM= 2021-05-27T23:59:22.541 INFO:teuthology.kill: smithi143.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAeTrNxzpcCoX7ypsYaneePSg7ist82aPHoigR9h/DFYsvlvAFlfp7XFos+qtPIBxhEsb9smRw4wOqCqZEYvtPbE0Bet0KXnVg/DqM5ADOdV4q6rkli1GJgrFExMJJh4Oi2e8dTLegqzMCJyi5j1Q8RsfoiX/QaE7vKe+8/6QqbnJ2CYK9/Bn8ECQKFamC69QkR+71TCnehtH3fv6OQ8E7pKArUoG8DF3Fl5sZ1b/z4WP/L2LidroODYlpkcj5qhI781Or9h9SQI6x587AmYTTbxCiSpxMoFAM1NsGHwqGT0sgx/ivT7/hbSaqHbFOhGAK1mHqokrAj07z0+t8Cxw7FugtXKtXocJ/95q7elrBWgkzeyLGjRIKy//OqPcJglEJhcZyPvFtDbuovsl1QnLxrbuMmHyVdaII7S4agozHIiMwunGiQWBZG9ZAxSW90b2fkzm8HTxguQISSkFLeZLj5ok/cfYnYTZU4m0dBm968bBrEO/z48gV7kIQhcD6asc= 2021-05-27T23:59:22.562 INFO:teuthology.kill:2021-05-27 23:59:22,562.562 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-27T23:59:22.567 INFO:teuthology.kill:2021-05-27 23:59:22,566.566 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-27T23:59:22.627 INFO:teuthology.kill:2021-05-27 23:59:22,627.627 INFO:teuthology.orchestra.console:Power off smithi031 2021-05-27T23:59:22.639 INFO:teuthology.kill:2021-05-27 23:59:22,639.639 INFO:teuthology.orchestra.console:Power off smithi143 2021-05-27T23:59:22.770 INFO:teuthology.kill:2021-05-27 23:59:22,770.770 INFO:teuthology.orchestra.console:Power off for smithi031 completed 2021-05-27T23:59:22.782 INFO:teuthology.kill:2021-05-27 23:59:22,782.782 INFO:teuthology.orchestra.console:Power off for smithi143 completed 2021-05-27T23:59:22.912 INFO:teuthology.kill:2021-05-27 23:59:22,911.911 INFO:teuthology.lock.ops:unlocked smithi031.front.sepia.ceph.com 2021-05-27T23:59:22.921 INFO:teuthology.kill:2021-05-27 23:59:22,921.921 INFO:teuthology.lock.ops:unlocked smithi143.front.sepia.ceph.com 2021-05-28T00:01:23.199 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-05-28T00:01:23.303 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-05-28T00:01:23.303 INFO:teuthology.nuke:Checking targets against current locks 2021-05-28T00:01:23.397 INFO:teuthology.nuke:Not nuking smithi143.front.sepia.ceph.com because description doesn't match 2021-05-28T00:01:23.397 INFO:teuthology.nuke:Not nuking smithi031.front.sepia.ceph.com because description doesn't match