2021-06-08T09:50:17.911 INFO:root:teuthology version: 1.1.0-f359b10d 2021-06-08T09:50:18.000 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi062.front.sepia.ceph.com' reimaging 2021-06-08T09:50:18.001 INFO:teuthology.lock.ops:Updating [ubuntu@smithi062.front.sepia.ceph.com]: reset os type and version on server 2021-06-08T09:50:18.001 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2021-06-08T09:50:18.017 INFO:teuthology.lock.ops:Node 'ubuntu@smithi062.front.sepia.ceph.com' reimaging is complete 2021-06-08T09:50:18.017 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi112.front.sepia.ceph.com' reimaging 2021-06-08T09:50:18.018 INFO:teuthology.lock.ops:Updating [ubuntu@smithi112.front.sepia.ceph.com]: reset os type and version on server 2021-06-08T09:50:18.018 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2021-06-08T09:50:18.032 INFO:teuthology.lock.ops:Node 'ubuntu@smithi112.front.sepia.ceph.com' reimaging is complete 2021-06-08T09:50:18.239 INFO:teuthology.provision.fog.smithi062:Scheduling deploy of ubuntu 20.04 2021-06-08T09:50:18.270 INFO:teuthology.provision.fog.smithi112:Scheduling deploy of ubuntu 20.04 2021-06-08T09:50:18.611 INFO:teuthology.orchestra.console:Power off smithi062 2021-06-08T09:50:18.667 INFO:teuthology.orchestra.console:Power off smithi112 2021-06-08T09:50:18.759 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2021-06-08T09:50:18.812 INFO:teuthology.orchestra.console:Power off for smithi112 completed 2021-06-08T09:50:18.864 INFO:teuthology.orchestra.console:Power on smithi062 2021-06-08T09:50:18.913 INFO:teuthology.orchestra.console:Power on smithi112 2021-06-08T09:50:23.117 INFO:teuthology.orchestra.console:Power on for smithi062 completed 2021-06-08T09:50:23.218 INFO:teuthology.provision.fog.smithi062:Waiting for deploy to finish 2021-06-08T09:50:27.354 INFO:teuthology.orchestra.console:Power on for smithi112 completed 2021-06-08T09:50:27.455 INFO:teuthology.provision.fog.smithi112:Waiting for deploy to finish 2021-06-08T09:53:13.797 ERROR:teuthology.orchestra.connection:Error authenticating with smithi112.front.sepia.ceph.com: Authentication failed. 2021-06-08T09:55:18.346 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-08T09:55:18.791 INFO:teuthology.provision.fog.smithi112:Node is ready 2021-06-08T09:55:18.797 INFO:teuthology.orchestra.run.smithi112.stdout:smithi112.front.sepia.ceph.com 2021-06-08T09:55:18.846 INFO:teuthology.orchestra.run.smithi112.stdout:172.21.15.112 smithi112.front.sepia.ceph.com smithi112 2021-06-08T09:55:19.060 INFO:teuthology.provision.fog.smithi112:Deploy complete! 2021-06-08T09:55:29.540 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-08T09:55:29.997 INFO:teuthology.provision.fog.smithi062:Node is ready 2021-06-08T09:55:30.046 INFO:teuthology.orchestra.run.smithi062.stdout:smithi062.front.sepia.ceph.com 2021-06-08T09:55:30.054 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62 smithi062.front.sepia.ceph.com smithi062 2021-06-08T09:55:30.265 INFO:teuthology.provision.fog.smithi062:Deploy complete! 2021-06-08T09:55:30.445 INFO:teuthology.lock.ops:Checking smithi062.front.sepia.ceph.com 2021-06-08T09:55:30.446 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-08T09:55:30.462 INFO:teuthology.lock.ops:Checking smithi112.front.sepia.ceph.com 2021-06-08T09:55:30.462 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-08T09:55:30.476 INFO:teuthology.lock.ops:Updating [smithi062.front.sepia.ceph.com]: set os type and version on server 2021-06-08T09:55:30.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-08T09:55:30.655 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2021-06-08T09:55:30.702 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="Ubuntu" 2021-06-08T09:55:30.703 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-06-08T09:55:30.703 INFO:teuthology.orchestra.run.smithi062.stdout:ID=ubuntu 2021-06-08T09:55:30.703 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE=debian 2021-06-08T09:55:30.704 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-06-08T09:55:30.704 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="20.04" 2021-06-08T09:55:30.704 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://www.ubuntu.com/" 2021-06-08T09:55:30.704 INFO:teuthology.orchestra.run.smithi062.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-06-08T09:55:30.705 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-06-08T09:55:30.705 INFO:teuthology.orchestra.run.smithi062.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-06-08T09:55:30.705 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_CODENAME=focal 2021-06-08T09:55:30.705 INFO:teuthology.orchestra.run.smithi062.stdout:UBUNTU_CODENAME=focal 2021-06-08T09:55:30.706 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2021-06-08T09:55:30.720 INFO:teuthology.lock.ops:Updating [smithi112.front.sepia.ceph.com]: set os type and version on server 2021-06-08T09:55:30.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-08T09:55:30.975 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2021-06-08T09:55:30.982 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="Ubuntu" 2021-06-08T09:55:30.982 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-06-08T09:55:30.982 INFO:teuthology.orchestra.run.smithi112.stdout:ID=ubuntu 2021-06-08T09:55:30.983 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE=debian 2021-06-08T09:55:30.983 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-06-08T09:55:30.983 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="20.04" 2021-06-08T09:55:30.983 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://www.ubuntu.com/" 2021-06-08T09:55:30.983 INFO:teuthology.orchestra.run.smithi112.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-06-08T09:55:30.984 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-06-08T09:55:30.984 INFO:teuthology.orchestra.run.smithi112.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-06-08T09:55:30.984 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_CODENAME=focal 2021-06-08T09:55:30.984 INFO:teuthology.orchestra.run.smithi112.stdout:UBUNTU_CODENAME=focal 2021-06-08T09:55:30.985 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2021-06-08T09:55:31.066 INFO:teuthology.dispatcher.supervisor:Running job 6158856 2021-06-08T09:55:31.066 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-06-08_04:17:01-fs-pacific-distro-basic-smithi/6158856 --name teuthology-2021-06-08_04:17:01-fs-pacific-distro-basic-smithi --description fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snapshots} -- /home/teuthworker/archive/teuthology-2021-06-08_04:17:01-fs-pacific-distro-basic-smithi/6158856/orig.config.yaml 2021-06-08T09:55:31.071 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-06-08_04:17:01-fs-pacific-distro-basic-smithi/6158856 2021-06-08T09:55:31.071 INFO:teuthology.dispatcher.supervisor:Job PID: 14763 2021-06-08T09:55:31.072 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-06-08T21:55:48.888 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-06-08T21:55:49.179 INFO:teuthology.kill:Killing Pids: {14763} 2021-06-08T21:55:50.498 INFO:teuthology.kill:Nuking machines: ['smithi062', 'smithi112'] 2021-06-08T21:55:51.448 INFO:teuthology.kill:2021-06-08 21:55:51,448.448 INFO:teuthology.nuke:targets: 2021-06-08T21:55:51.449 INFO:teuthology.kill: smithi062.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSDSvh3PlVA+yiTuyO+57pbGemvP0n/ejY2FPjyWS3fV9f7ACz0LiHDYQ/QQ0bFFTNywwjn9AER+4sfVSxU5Jy76PVXvLm/I5mtOfD7QdLJnVoxpSeFFClTMuQenBS4CbjsnXBqyFqxvm0RyHm28SNtXC1TBeyVzXkaRkgtkNfIxNeY3C24Yl1+qNirIQGMroCS4FcQ4yvBgRYtylFDq9fyLHivoc0sBfTrs+hheqppM8R8MBk8aHSd6Um1V4nJE5Gqk6DZwiXORgiZlggIkYdRgnt2fon65s+gxZRMuElRxmJkW0qhGcOZnQhcDvX6NvGNKb7w1qIF53y8nR5oX6xGwG2UQEX7gGS3rKUsJhktAXsZSn6JjaPxyDEdiueIFfiXdmw4HHTlOQP68A3eYkuiJR5WMq89M3vhN2E9yla05VueXKIbjGRLQfNKH8nUE8HS9zzRkfQ7j+h8AamJB2Cta8r9+ZsOcNndLIjXvxn3EVxITyhRo1PkV6xTtVi+E8= 2021-06-08T21:55:51.450 INFO:teuthology.kill: smithi112.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnq3CMziXeGa4E78HVYASUL0356A+h4w/D77xx7qk09nUWZBg/0mHu766M6dRwPVmEjvQaFT8R8yOshoU/auyPBqPUhMIHQcTg0EvzSRRaaNaCtKCiSM7gNveysv3U94jFUyX5/Be2sd+co7mJeswGrxAVwmStRUfxk8wdIYFA4n0pdrkveBNoHiZvdVLqHphHOm5R/DVLr9nQKjMv+U/edYgT7ya4k4278CvMHFL4WZ/XnQwyct656XyXc5eL3d6AaWSx7yPF1LKbR+tHIgtyX7aLGdapE/h8fWv+Eh8CuWNAQOpWCjq9WEWDD9gH1eh8aZ4blX0db+izDLdaMPsG4gYY9aLp6ZAWqQJQzwkNxHvZcX4PTnk1rPM/81H4vBrNkfxfwPdtPMGjEnDEJ8+vNMEjLtlGEBH3cvIp4piqdybaLem3crahXKMo7zTmq7qZWFNSHc1zX9lDTNkv7DZVcG2dQ1sbjQHRttbWPikccB9lgn/KBjyX74f2pvi4wEU= 2021-06-08T21:55:51.450 INFO:teuthology.kill:2021-06-08 21:55:51,449.449 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-08T21:55:51.451 INFO:teuthology.kill:2021-06-08 21:55:51,451.451 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-08T21:55:51.511 INFO:teuthology.kill:2021-06-08 21:55:51,511.511 INFO:teuthology.orchestra.console:Power off smithi112 2021-06-08T21:55:51.524 INFO:teuthology.kill:2021-06-08 21:55:51,524.524 INFO:teuthology.orchestra.console:Power off smithi062 2021-06-08T21:55:59.880 INFO:teuthology.kill:2021-06-08 21:55:59,879.879 INFO:teuthology.orchestra.console:Power off for smithi112 completed 2021-06-08T21:55:59.886 INFO:teuthology.kill:2021-06-08 21:55:59,886.886 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2021-06-08T21:56:00.112 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2021-06-08T21:56:00.113 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2021-06-08T21:56:00.113 INFO:teuthology.misc:Compressing logs... 2021-06-08T21:56:00.114 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-08T21:57:00.118 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-06-08T21:57:00.185 INFO:teuthology.kill:No teuthology processes running 2021-06-08T21:57:01.292 INFO:teuthology.kill:Nuking machines: ['smithi062', 'smithi112'] 2021-06-08T21:57:02.229 INFO:teuthology.kill:2021-06-08 21:57:02,228.228 INFO:teuthology.nuke:targets: 2021-06-08T21:57:02.229 INFO:teuthology.kill: smithi062.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSDSvh3PlVA+yiTuyO+57pbGemvP0n/ejY2FPjyWS3fV9f7ACz0LiHDYQ/QQ0bFFTNywwjn9AER+4sfVSxU5Jy76PVXvLm/I5mtOfD7QdLJnVoxpSeFFClTMuQenBS4CbjsnXBqyFqxvm0RyHm28SNtXC1TBeyVzXkaRkgtkNfIxNeY3C24Yl1+qNirIQGMroCS4FcQ4yvBgRYtylFDq9fyLHivoc0sBfTrs+hheqppM8R8MBk8aHSd6Um1V4nJE5Gqk6DZwiXORgiZlggIkYdRgnt2fon65s+gxZRMuElRxmJkW0qhGcOZnQhcDvX6NvGNKb7w1qIF53y8nR5oX6xGwG2UQEX7gGS3rKUsJhktAXsZSn6JjaPxyDEdiueIFfiXdmw4HHTlOQP68A3eYkuiJR5WMq89M3vhN2E9yla05VueXKIbjGRLQfNKH8nUE8HS9zzRkfQ7j+h8AamJB2Cta8r9+ZsOcNndLIjXvxn3EVxITyhRo1PkV6xTtVi+E8= 2021-06-08T21:57:02.230 INFO:teuthology.kill: smithi112.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnq3CMziXeGa4E78HVYASUL0356A+h4w/D77xx7qk09nUWZBg/0mHu766M6dRwPVmEjvQaFT8R8yOshoU/auyPBqPUhMIHQcTg0EvzSRRaaNaCtKCiSM7gNveysv3U94jFUyX5/Be2sd+co7mJeswGrxAVwmStRUfxk8wdIYFA4n0pdrkveBNoHiZvdVLqHphHOm5R/DVLr9nQKjMv+U/edYgT7ya4k4278CvMHFL4WZ/XnQwyct656XyXc5eL3d6AaWSx7yPF1LKbR+tHIgtyX7aLGdapE/h8fWv+Eh8CuWNAQOpWCjq9WEWDD9gH1eh8aZ4blX0db+izDLdaMPsG4gYY9aLp6ZAWqQJQzwkNxHvZcX4PTnk1rPM/81H4vBrNkfxfwPdtPMGjEnDEJ8+vNMEjLtlGEBH3cvIp4piqdybaLem3crahXKMo7zTmq7qZWFNSHc1zX9lDTNkv7DZVcG2dQ1sbjQHRttbWPikccB9lgn/KBjyX74f2pvi4wEU= 2021-06-08T21:57:02.250 INFO:teuthology.kill:2021-06-08 21:57:02,250.250 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-08T21:57:02.253 INFO:teuthology.kill:2021-06-08 21:57:02,253.253 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-08T21:57:02.313 INFO:teuthology.kill:2021-06-08 21:57:02,313.313 INFO:teuthology.orchestra.console:Power off smithi062 2021-06-08T21:57:02.337 INFO:teuthology.kill:2021-06-08 21:57:02,337.337 INFO:teuthology.orchestra.console:Power off smithi112 2021-06-08T21:57:02.469 INFO:teuthology.kill:2021-06-08 21:57:02,468.468 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2021-06-08T21:57:02.481 INFO:teuthology.kill:2021-06-08 21:57:02,481.481 INFO:teuthology.orchestra.console:Power off for smithi112 completed 2021-06-08T21:57:02.615 INFO:teuthology.kill:2021-06-08 21:57:02,614.614 INFO:teuthology.lock.ops:unlocked smithi062.front.sepia.ceph.com 2021-06-08T21:57:02.625 INFO:teuthology.kill:2021-06-08 21:57:02,625.625 INFO:teuthology.lock.ops:unlocked smithi112.front.sepia.ceph.com 2021-06-08T21:59:02.822 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-06-08T21:59:02.927 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-06-08T21:59:02.928 INFO:teuthology.nuke:Checking targets against current locks 2021-06-08T21:59:03.001 INFO:teuthology.nuke:Not nuking smithi112.front.sepia.ceph.com because description doesn't match 2021-06-08T21:59:03.002 INFO:teuthology.nuke:Not nuking smithi062.front.sepia.ceph.com because description doesn't match