2021-06-01T07:16:53.831 INFO:root:teuthology version: 1.1.0-f359b10d 2021-06-01T07:16:53.929 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi033.front.sepia.ceph.com' reimaging 2021-06-01T07:16:53.931 INFO:teuthology.lock.ops:Updating [ubuntu@smithi033.front.sepia.ceph.com]: reset os type and version on server 2021-06-01T07:16:53.931 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2021-06-01T07:16:53.951 INFO:teuthology.lock.ops:Node 'ubuntu@smithi033.front.sepia.ceph.com' reimaging is complete 2021-06-01T07:16:53.951 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi073.front.sepia.ceph.com' reimaging 2021-06-01T07:16:53.952 INFO:teuthology.lock.ops:Updating [ubuntu@smithi073.front.sepia.ceph.com]: reset os type and version on server 2021-06-01T07:16:53.952 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server 2021-06-01T07:16:53.971 INFO:teuthology.lock.ops:Node 'ubuntu@smithi073.front.sepia.ceph.com' reimaging is complete 2021-06-01T07:16:54.179 INFO:teuthology.provision.fog.smithi033:Scheduling deploy of centos 8.2 2021-06-01T07:16:54.191 INFO:teuthology.provision.fog.smithi073:Scheduling deploy of centos 8.2 2021-06-01T07:16:54.698 INFO:teuthology.orchestra.console:Power off smithi073 2021-06-01T07:16:54.712 INFO:teuthology.orchestra.console:Power off smithi033 2021-06-01T07:17:03.072 INFO:teuthology.orchestra.console:Power off for smithi073 completed 2021-06-01T07:17:03.079 INFO:teuthology.orchestra.console:Power off for smithi033 completed 2021-06-01T07:17:03.173 INFO:teuthology.orchestra.console:Power on smithi073 2021-06-01T07:17:03.181 INFO:teuthology.orchestra.console:Power on smithi033 2021-06-01T07:17:11.618 INFO:teuthology.orchestra.console:Power on for smithi073 completed 2021-06-01T07:17:11.667 INFO:teuthology.orchestra.console:Power on for smithi033 completed 2021-06-01T07:17:11.720 INFO:teuthology.provision.fog.smithi073:Waiting for deploy to finish 2021-06-01T07:17:11.768 INFO:teuthology.provision.fog.smithi033:Waiting for deploy to finish 2021-06-01T07:21:19.606 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-01T07:21:19.961 INFO:teuthology.provision.fog.smithi033:Node is ready 2021-06-01T07:21:19.977 INFO:teuthology.orchestra.run.smithi033.stdout:smithi033.front.sepia.ceph.com 2021-06-01T07:21:20.034 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33 smithi033.front.sepia.ceph.com smithi033 2021-06-01T07:21:20.364 INFO:teuthology.provision.fog.smithi033:Deploy complete! 2021-06-01T07:21:43.877 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-01T07:21:44.183 INFO:teuthology.provision.fog.smithi073:Node is ready 2021-06-01T07:21:44.199 INFO:teuthology.orchestra.run.smithi073.stdout:smithi073.front.sepia.ceph.com 2021-06-01T07:21:44.255 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73 smithi073.front.sepia.ceph.com smithi073 2021-06-01T07:21:44.604 INFO:teuthology.provision.fog.smithi073:Deploy complete! 2021-06-01T07:21:44.728 INFO:teuthology.lock.ops:Checking smithi033.front.sepia.ceph.com 2021-06-01T07:21:44.730 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-01T07:21:44.747 INFO:teuthology.lock.ops:Checking smithi073.front.sepia.ceph.com 2021-06-01T07:21:44.748 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-01T07:21:44.765 INFO:teuthology.lock.ops:Updating [smithi033.front.sepia.ceph.com]: set os type and version on server 2021-06-01T07:21:44.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-01T07:21:45.047 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2021-06-01T07:21:45.102 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="CentOS Linux" 2021-06-01T07:21:45.103 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="8 (Core)" 2021-06-01T07:21:45.103 INFO:teuthology.orchestra.run.smithi033.stdout:ID="centos" 2021-06-01T07:21:45.103 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE="rhel fedora" 2021-06-01T07:21:45.103 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="8" 2021-06-01T07:21:45.104 INFO:teuthology.orchestra.run.smithi033.stdout:PLATFORM_ID="platform:el8" 2021-06-01T07:21:45.104 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-06-01T07:21:45.104 INFO:teuthology.orchestra.run.smithi033.stdout:ANSI_COLOR="0;31" 2021-06-01T07:21:45.104 INFO:teuthology.orchestra.run.smithi033.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-01T07:21:45.105 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://www.centos.org/" 2021-06-01T07:21:45.105 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-01T07:21:45.105 INFO:teuthology.orchestra.run.smithi033.stdout: 2021-06-01T07:21:45.105 INFO:teuthology.orchestra.run.smithi033.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-01T07:21:45.105 INFO:teuthology.orchestra.run.smithi033.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-01T07:21:45.106 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-06-01T07:21:45.106 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-06-01T07:21:45.106 INFO:teuthology.orchestra.run.smithi033.stdout: 2021-06-01T07:21:45.107 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2021-06-01T07:21:45.122 INFO:teuthology.lock.ops:Updating [smithi073.front.sepia.ceph.com]: set os type and version on server 2021-06-01T07:21:45.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-01T07:21:45.395 INFO:teuthology.orchestra.run.smithi073.stdout:x86_64 2021-06-01T07:21:45.450 INFO:teuthology.orchestra.run.smithi073.stdout:NAME="CentOS Linux" 2021-06-01T07:21:45.450 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION="8 (Core)" 2021-06-01T07:21:45.451 INFO:teuthology.orchestra.run.smithi073.stdout:ID="centos" 2021-06-01T07:21:45.451 INFO:teuthology.orchestra.run.smithi073.stdout:ID_LIKE="rhel fedora" 2021-06-01T07:21:45.452 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION_ID="8" 2021-06-01T07:21:45.452 INFO:teuthology.orchestra.run.smithi073.stdout:PLATFORM_ID="platform:el8" 2021-06-01T07:21:45.452 INFO:teuthology.orchestra.run.smithi073.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-06-01T07:21:45.453 INFO:teuthology.orchestra.run.smithi073.stdout:ANSI_COLOR="0;31" 2021-06-01T07:21:45.454 INFO:teuthology.orchestra.run.smithi073.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-01T07:21:45.454 INFO:teuthology.orchestra.run.smithi073.stdout:HOME_URL="https://www.centos.org/" 2021-06-01T07:21:45.454 INFO:teuthology.orchestra.run.smithi073.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-01T07:21:45.455 INFO:teuthology.orchestra.run.smithi073.stdout: 2021-06-01T07:21:45.455 INFO:teuthology.orchestra.run.smithi073.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-01T07:21:45.456 INFO:teuthology.orchestra.run.smithi073.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-01T07:21:45.456 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-06-01T07:21:45.456 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-06-01T07:21:45.457 INFO:teuthology.orchestra.run.smithi073.stdout: 2021-06-01T07:21:45.458 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server 2021-06-01T07:21:45.549 INFO:teuthology.dispatcher.supervisor:Running job 6144675 2021-06-01T07:21:45.550 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-06-01_04:17:03-fs-pacific-distro-basic-smithi/6144675 --name teuthology-2021-06-01_04:17:03-fs-pacific-distro-basic-smithi --description fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} -- /home/teuthworker/archive/teuthology-2021-06-01_04:17:03-fs-pacific-distro-basic-smithi/6144675/orig.config.yaml 2021-06-01T07:21:45.554 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-06-01_04:17:03-fs-pacific-distro-basic-smithi/6144675 2021-06-01T07:21:45.555 INFO:teuthology.dispatcher.supervisor:Job PID: 28369 2021-06-01T07:21:45.555 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-06-01T19:22:33.223 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-06-01T19:22:33.383 INFO:teuthology.kill:Killing Pids: {28369} 2021-06-01T19:22:34.417 INFO:teuthology.kill:Nuking machines: ['smithi033', 'smithi073'] 2021-06-01T19:22:35.254 INFO:teuthology.kill:2021-06-01 19:22:35,254.254 INFO:teuthology.nuke:targets: 2021-06-01T19:22:35.255 INFO:teuthology.kill: smithi033.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCz8EDfeTo/baRRrIBO05VcX9HCQp+QFrYaQq/COWaG9Buv1xy6MbalMu/5eU/iGrbAXUXRwmPM9FJeeJR+qTfEo7PH43FCrgEpAnWN2Hu2QQbsdNAiHHyNmO/RyoTJZbz4ZgIOrzFfMOP/cNHUtZU3MQadanB/MYXHl/pSlg25+rZp5pEm8ownHXOmn2xFyHdRLpQ4S5PNLDnrfbyH16GOt6elXdKWLophfMN5d9LEuR9kdHre4s7vozJWWeCweUN1UeJlnp+OAcZ5gOZYHX86haBPl1rn/W6GSfQlScff+tx1s4hSnIghs7b9JpjQw2ewNvqSjTTtP3H6ecXDeHK45B1SQfsgcPtxx/HNISZhFluGdPwM0PJCRrAk0KzVOHGjRmdd80EeniIHodGU3cZmKku8TJ/4f9slWxH+5PzXU0Mj12xVoHa7CfnrPLZfPpPo9tGBHqqe6ohB/fQ46bWu3SdT6hzdkGMIkBQFLTTCQTVMOQSQTHk7nZGp6Gg8bwU= 2021-06-01T19:22:35.255 INFO:teuthology.kill: smithi073.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwlQ/uHO6eqz02HsEFbpjO2a9OOc5PbIzQZKv9P08w2yX2tF83bG1e2g0O0QbWgYnl2XDZIqfInI83V2vKw+YhSn6LldH/R+7ao7lFwUKQ/D4z7RNeJWPFlxo1S6wEkGAG/AWY2LsfS5hRdYMGIUf+97GKUcayHAedThwjKKVsIaQ+Y3blNMu/PdJ20cmrCeZPEEYtQMjWyQTbw25XPZ5AWPr6sZltwS+5o0x2kSpp7UJFMDACaCJ1U6vId1kj/9fSPzshkpjDrFuQlpWf4TOyMC2/NSfmf6UFfJTsKdZLDuf4zoKbRgFhVUQOaDaUx4pqySRyU2kS8t29t5r+EHQMXRJTSfyFHU0nuYsi6Ry6k0kCdPJAmyTGyA3Lvd3p/1SAuGiihG5LANd9Sgng6BZWTBBpDi3N2iYo4Au93I687rqGigl2MLfMaz2LgAAY/cQopU8/luCp+M3+BiXRCczhIZqqTSGtX7IqqIoKROOvzok68lpO2C0kGhF0/IDDrFU= 2021-06-01T19:22:35.256 INFO:teuthology.kill:2021-06-01 19:22:35,254.254 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-01T19:22:35.256 INFO:teuthology.kill:2021-06-01 19:22:35,256.256 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-01T19:22:35.313 INFO:teuthology.kill:2021-06-01 19:22:35,313.313 INFO:teuthology.orchestra.console:Power off smithi033 2021-06-01T19:22:35.326 INFO:teuthology.kill:2021-06-01 19:22:35,325.325 INFO:teuthology.orchestra.console:Power off smithi073 2021-06-01T19:22:43.682 INFO:teuthology.kill:2021-06-01 19:22:43,682.682 INFO:teuthology.orchestra.console:Power off for smithi033 completed 2021-06-01T19:22:47.802 INFO:teuthology.kill:2021-06-01 19:22:47,801.801 INFO:teuthology.orchestra.console:Power off for smithi073 completed 2021-06-01T19:22:48.008 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-06-01T19:22:48.008 INFO:teuthology.task.internal:roles: ubuntu@smithi073.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2021-06-01T19:22:48.009 INFO:teuthology.misc:Compressing logs... 2021-06-01T19:22:48.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-01T19:23:22.699 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-06-01T19:23:22.754 INFO:teuthology.kill:No teuthology processes running 2021-06-01T19:23:23.676 INFO:teuthology.kill:Nuking machines: ['smithi033', 'smithi073'] 2021-06-01T19:23:24.504 INFO:teuthology.kill:2021-06-01 19:23:24,504.504 INFO:teuthology.nuke:targets: 2021-06-01T19:23:24.505 INFO:teuthology.kill: smithi033.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCz8EDfeTo/baRRrIBO05VcX9HCQp+QFrYaQq/COWaG9Buv1xy6MbalMu/5eU/iGrbAXUXRwmPM9FJeeJR+qTfEo7PH43FCrgEpAnWN2Hu2QQbsdNAiHHyNmO/RyoTJZbz4ZgIOrzFfMOP/cNHUtZU3MQadanB/MYXHl/pSlg25+rZp5pEm8ownHXOmn2xFyHdRLpQ4S5PNLDnrfbyH16GOt6elXdKWLophfMN5d9LEuR9kdHre4s7vozJWWeCweUN1UeJlnp+OAcZ5gOZYHX86haBPl1rn/W6GSfQlScff+tx1s4hSnIghs7b9JpjQw2ewNvqSjTTtP3H6ecXDeHK45B1SQfsgcPtxx/HNISZhFluGdPwM0PJCRrAk0KzVOHGjRmdd80EeniIHodGU3cZmKku8TJ/4f9slWxH+5PzXU0Mj12xVoHa7CfnrPLZfPpPo9tGBHqqe6ohB/fQ46bWu3SdT6hzdkGMIkBQFLTTCQTVMOQSQTHk7nZGp6Gg8bwU= 2021-06-01T19:23:24.506 INFO:teuthology.kill: smithi073.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwlQ/uHO6eqz02HsEFbpjO2a9OOc5PbIzQZKv9P08w2yX2tF83bG1e2g0O0QbWgYnl2XDZIqfInI83V2vKw+YhSn6LldH/R+7ao7lFwUKQ/D4z7RNeJWPFlxo1S6wEkGAG/AWY2LsfS5hRdYMGIUf+97GKUcayHAedThwjKKVsIaQ+Y3blNMu/PdJ20cmrCeZPEEYtQMjWyQTbw25XPZ5AWPr6sZltwS+5o0x2kSpp7UJFMDACaCJ1U6vId1kj/9fSPzshkpjDrFuQlpWf4TOyMC2/NSfmf6UFfJTsKdZLDuf4zoKbRgFhVUQOaDaUx4pqySRyU2kS8t29t5r+EHQMXRJTSfyFHU0nuYsi6Ry6k0kCdPJAmyTGyA3Lvd3p/1SAuGiihG5LANd9Sgng6BZWTBBpDi3N2iYo4Au93I687rqGigl2MLfMaz2LgAAY/cQopU8/luCp+M3+BiXRCczhIZqqTSGtX7IqqIoKROOvzok68lpO2C0kGhF0/IDDrFU= 2021-06-01T19:23:24.524 INFO:teuthology.kill:2021-06-01 19:23:24,524.524 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-01T19:23:24.531 INFO:teuthology.kill:2021-06-01 19:23:24,531.531 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-01T19:23:24.585 INFO:teuthology.kill:2021-06-01 19:23:24,585.585 INFO:teuthology.orchestra.console:Power off smithi033 2021-06-01T19:23:24.597 INFO:teuthology.kill:2021-06-01 19:23:24,597.597 INFO:teuthology.orchestra.console:Power off smithi073 2021-06-01T19:23:24.724 INFO:teuthology.kill:2021-06-01 19:23:24,724.724 INFO:teuthology.orchestra.console:Power off for smithi033 completed 2021-06-01T19:23:24.737 INFO:teuthology.kill:2021-06-01 19:23:24,737.737 INFO:teuthology.orchestra.console:Power off for smithi073 completed 2021-06-01T19:23:24.866 INFO:teuthology.kill:2021-06-01 19:23:24,865.865 INFO:teuthology.lock.ops:unlocked smithi033.front.sepia.ceph.com 2021-06-01T19:23:24.874 INFO:teuthology.kill:2021-06-01 19:23:24,874.874 INFO:teuthology.lock.ops:unlocked smithi073.front.sepia.ceph.com 2021-06-01T19:25:25.138 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15