2021-05-14T03:23:46.360 INFO:root:teuthology version: 1.1.0-19220a3b 2021-05-14T03:23:46.480 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi116.front.sepia.ceph.com' reimaging 2021-05-14T03:23:46.482 INFO:teuthology.lock.ops:Updating [ubuntu@smithi116.front.sepia.ceph.com]: reset os type and version on server 2021-05-14T03:23:46.482 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2021-05-14T03:23:46.497 INFO:teuthology.lock.ops:Node 'ubuntu@smithi116.front.sepia.ceph.com' reimaging is complete 2021-05-14T03:23:46.498 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi154.front.sepia.ceph.com' reimaging 2021-05-14T03:23:46.498 INFO:teuthology.lock.ops:Updating [ubuntu@smithi154.front.sepia.ceph.com]: reset os type and version on server 2021-05-14T03:23:46.498 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2021-05-14T03:23:46.520 INFO:teuthology.lock.ops:Node 'ubuntu@smithi154.front.sepia.ceph.com' reimaging is complete 2021-05-14T03:23:46.704 INFO:teuthology.provision.fog.smithi116:Scheduling deploy of ubuntu 20.04 2021-05-14T03:23:46.714 INFO:teuthology.provision.fog.smithi154:Scheduling deploy of ubuntu 20.04 2021-05-14T03:23:47.026 INFO:teuthology.orchestra.console:Power off smithi116 2021-05-14T03:23:47.041 INFO:teuthology.orchestra.console:Power off smithi154 2021-05-14T03:23:55.395 INFO:teuthology.orchestra.console:Power off for smithi116 completed 2021-05-14T03:23:55.401 INFO:teuthology.orchestra.console:Power off for smithi154 completed 2021-05-14T03:23:55.497 INFO:teuthology.orchestra.console:Power on smithi116 2021-05-14T03:23:55.505 INFO:teuthology.orchestra.console:Power on smithi154 2021-05-14T03:24:03.864 INFO:teuthology.orchestra.console:Power on for smithi116 completed 2021-05-14T03:24:03.870 INFO:teuthology.orchestra.console:Power on for smithi154 completed 2021-05-14T03:24:03.966 INFO:teuthology.provision.fog.smithi116:Waiting for deploy to finish 2021-05-14T03:24:03.970 INFO:teuthology.provision.fog.smithi154:Waiting for deploy to finish 2021-05-14T03:26:50.442 ERROR:teuthology.orchestra.connection:Error authenticating with smithi116.front.sepia.ceph.com: Authentication failed. 2021-05-14T03:29:07.598 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-14T03:29:08.049 INFO:teuthology.provision.fog.smithi154:Node is ready 2021-05-14T03:29:08.056 INFO:teuthology.orchestra.run.smithi154.stdout:smithi154.front.sepia.ceph.com 2021-05-14T03:29:08.064 INFO:teuthology.orchestra.run.smithi154.stdout:172.21.15.154 smithi154.front.sepia.ceph.com smithi154 2021-05-14T03:29:08.305 INFO:teuthology.provision.fog.smithi154:Deploy complete! 2021-05-14T03:29:35.593 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-14T03:29:36.084 INFO:teuthology.provision.fog.smithi116:Node is ready 2021-05-14T03:29:36.091 INFO:teuthology.orchestra.run.smithi116.stdout:smithi116.front.sepia.ceph.com 2021-05-14T03:29:36.099 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116 smithi116.front.sepia.ceph.com smithi116 2021-05-14T03:29:36.332 INFO:teuthology.provision.fog.smithi116:Deploy complete! 2021-05-14T03:29:36.488 INFO:teuthology.lock.ops:Checking smithi116.front.sepia.ceph.com 2021-05-14T03:29:36.489 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-14T03:29:36.505 INFO:teuthology.lock.ops:Checking smithi154.front.sepia.ceph.com 2021-05-14T03:29:36.505 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-14T03:29:36.520 INFO:teuthology.lock.ops:Updating [smithi116.front.sepia.ceph.com]: set os type and version on server 2021-05-14T03:29:36.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-14T03:29:36.702 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2021-05-14T03:29:36.752 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="Ubuntu" 2021-05-14T03:29:36.753 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-05-14T03:29:36.753 INFO:teuthology.orchestra.run.smithi116.stdout:ID=ubuntu 2021-05-14T03:29:36.753 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE=debian 2021-05-14T03:29:36.754 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-05-14T03:29:36.755 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="20.04" 2021-05-14T03:29:36.755 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://www.ubuntu.com/" 2021-05-14T03:29:36.756 INFO:teuthology.orchestra.run.smithi116.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-05-14T03:29:36.757 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-05-14T03:29:36.757 INFO:teuthology.orchestra.run.smithi116.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-05-14T03:29:36.758 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_CODENAME=focal 2021-05-14T03:29:36.758 INFO:teuthology.orchestra.run.smithi116.stdout:UBUNTU_CODENAME=focal 2021-05-14T03:29:36.759 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2021-05-14T03:29:36.775 INFO:teuthology.lock.ops:Updating [smithi154.front.sepia.ceph.com]: set os type and version on server 2021-05-14T03:29:36.775 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-14T03:29:37.030 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2021-05-14T03:29:37.079 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="Ubuntu" 2021-05-14T03:29:37.080 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-05-14T03:29:37.080 INFO:teuthology.orchestra.run.smithi154.stdout:ID=ubuntu 2021-05-14T03:29:37.081 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE=debian 2021-05-14T03:29:37.081 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-05-14T03:29:37.082 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="20.04" 2021-05-14T03:29:37.082 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://www.ubuntu.com/" 2021-05-14T03:29:37.083 INFO:teuthology.orchestra.run.smithi154.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-05-14T03:29:37.083 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-05-14T03:29:37.083 INFO:teuthology.orchestra.run.smithi154.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-05-14T03:29:37.083 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_CODENAME=focal 2021-05-14T03:29:37.084 INFO:teuthology.orchestra.run.smithi154.stdout:UBUNTU_CODENAME=focal 2021-05-14T03:29:37.084 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2021-05-14T03:29:37.168 INFO:teuthology.dispatcher.supervisor:Running job 6114622 2021-05-14T03:29:37.168 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/virtualenv/bin/teuthology -v --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2021-05-14_03:03:45-fs-wip-pdonnell-testing-20210511.232042-distro-basic-smithi/6114622 --name pdonnell-2021-05-14_03:03:45-fs-wip-pdonnell-testing-20210511.232042-distro-basic-smithi --description fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}} -- /home/teuthworker/archive/pdonnell-2021-05-14_03:03:45-fs-wip-pdonnell-testing-20210511.232042-distro-basic-smithi/6114622/orig.config.yaml 2021-05-14T03:29:37.172 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/pdonnell-2021-05-14_03:03:45-fs-wip-pdonnell-testing-20210511.232042-distro-basic-smithi/6114622 2021-05-14T03:29:37.173 INFO:teuthology.dispatcher.supervisor:Job PID: 21556 2021-05-14T03:29:37.173 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-14T15:29:51.600 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-14T15:29:51.741 INFO:teuthology.kill:Killing Pids: {21556} 2021-05-14T15:29:52.753 INFO:teuthology.kill:Nuking machines: ['smithi116', 'smithi154'] 2021-05-14T15:29:53.649 INFO:teuthology.kill:2021-05-14 15:29:53,649.649 INFO:teuthology.nuke:targets: 2021-05-14T15:29:53.650 INFO:teuthology.kill: smithi116.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCii+JCBuqow3Zb4BhKPRqiRQhiaN5rWbVJi3twQ4xR0HYOvY6dOE24I//29mEUc9tcMB5p+vRKkAy0DbzuCToebJc5laaufzVDw2FQdef5m0MWUb3qWwWcMr8tuqNHeeWTYb3rg7YxeN7mmjCCs2gZ98E5i05uUQXY9a/qgGKd3/2nY7zmEsqXLZ184BpjEO7GzsF3oUt08JLQ/Fs7EmGlc25UAmCBJ4QfuUJIndTnuCRaPLzItxVzu/cI2sbmj5U/yaFG68McMh2ATseeDdgvW87/mzd1Ef+he0i9tevXTJaOB3WBsYVJ3Q4GwMzzf5dpzFMAO3IkWJQV+8oMtImoOfmlFcTmecy9N55zd0phszUt0wIc1uK/3Nsa+HfFOUjnzzbcal+8zz+PeW5rRUNwX05sSfE64wW2NvF7cUl+jI+vI+YM1BFJMifGAW52eRX1XyAN+/w8Pa4Fgw8DJ5WvolLon2EmOp1GfZSgdCXTpz54coPzuYlOfNkbUZ5NWgE= 2021-05-14T15:29:53.650 INFO:teuthology.kill: smithi154.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCiP88LrRnPNmERfhFCnDOS9fSb88JZX+bbgyBGlCeOVOKdXtKAMiVYsT9MS7/bw8jAD9NZ0BFiKR/i2FNm+6KXI1sNK+GluHOGDPFgQhKdjikitV8Al4pX5GzPeFG1OWWBJ3hgnXxivTpyrQSdqDubEu+CLs3jhUjWCXq2B9di2+rP1Ed4iMpCpLQhg+HwD4Jov8oJaMffxBjYmBpCaOGA+M47lCfZ4k5gfBbBrIAMu41/RgUCbS1m2Li3qRfGzm804FMHz62UohegB9uH0S6Co3KoH2CGn5LY7mdK4E1acS6nOg7xohwy4UlqUaUvw6wvHzZxuNnV8YWkz8CfVc9cEJsvmdII1+doe1QYkGxyhiI5Bf2cKgNJYxr539DwMKDMY2b0HpUFCTJYXpcYYJ8RhBroxsGGtt9pyPu0KZf6lScVPtyQMBzy0bODHRjkkS4q3F4xWEeBaMnh7BI6Vcta/uifB/WsoGRnRnjlZLQOhRgMSOq+nxINKpqDMlazvlk= 2021-05-14T15:29:53.650 INFO:teuthology.kill:2021-05-14 15:29:53,649.649 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-14T15:29:53.651 INFO:teuthology.kill:2021-05-14 15:29:53,651.651 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-14T15:29:53.702 INFO:teuthology.kill:2021-05-14 15:29:53,701.701 INFO:teuthology.orchestra.console:Power off smithi116 2021-05-14T15:29:53.714 INFO:teuthology.kill:2021-05-14 15:29:53,714.714 INFO:teuthology.orchestra.console:Power off smithi154 2021-05-14T15:30:02.074 INFO:teuthology.kill:2021-05-14 15:30:02,073.073 INFO:teuthology.orchestra.console:Power off for smithi154 completed 2021-05-14T15:30:06.181 INFO:teuthology.kill:2021-05-14 15:30:06,180.180 INFO:teuthology.orchestra.console:Power off for smithi116 completed 2021-05-14T15:30:06.388 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2021-05-14T15:30:06.389 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2021-05-14T15:30:06.389 INFO:teuthology.misc:Compressing logs... 2021-05-14T15:30:06.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-14T15:30:40.975 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-14T15:30:41.038 INFO:teuthology.kill:No teuthology processes running 2021-05-14T15:30:42.067 INFO:teuthology.kill:Nuking machines: ['smithi116', 'smithi154'] 2021-05-14T15:30:42.953 INFO:teuthology.kill:2021-05-14 15:30:42,952.952 INFO:teuthology.nuke:targets: 2021-05-14T15:30:42.954 INFO:teuthology.kill: smithi116.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCii+JCBuqow3Zb4BhKPRqiRQhiaN5rWbVJi3twQ4xR0HYOvY6dOE24I//29mEUc9tcMB5p+vRKkAy0DbzuCToebJc5laaufzVDw2FQdef5m0MWUb3qWwWcMr8tuqNHeeWTYb3rg7YxeN7mmjCCs2gZ98E5i05uUQXY9a/qgGKd3/2nY7zmEsqXLZ184BpjEO7GzsF3oUt08JLQ/Fs7EmGlc25UAmCBJ4QfuUJIndTnuCRaPLzItxVzu/cI2sbmj5U/yaFG68McMh2ATseeDdgvW87/mzd1Ef+he0i9tevXTJaOB3WBsYVJ3Q4GwMzzf5dpzFMAO3IkWJQV+8oMtImoOfmlFcTmecy9N55zd0phszUt0wIc1uK/3Nsa+HfFOUjnzzbcal+8zz+PeW5rRUNwX05sSfE64wW2NvF7cUl+jI+vI+YM1BFJMifGAW52eRX1XyAN+/w8Pa4Fgw8DJ5WvolLon2EmOp1GfZSgdCXTpz54coPzuYlOfNkbUZ5NWgE= 2021-05-14T15:30:42.954 INFO:teuthology.kill: smithi154.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCiP88LrRnPNmERfhFCnDOS9fSb88JZX+bbgyBGlCeOVOKdXtKAMiVYsT9MS7/bw8jAD9NZ0BFiKR/i2FNm+6KXI1sNK+GluHOGDPFgQhKdjikitV8Al4pX5GzPeFG1OWWBJ3hgnXxivTpyrQSdqDubEu+CLs3jhUjWCXq2B9di2+rP1Ed4iMpCpLQhg+HwD4Jov8oJaMffxBjYmBpCaOGA+M47lCfZ4k5gfBbBrIAMu41/RgUCbS1m2Li3qRfGzm804FMHz62UohegB9uH0S6Co3KoH2CGn5LY7mdK4E1acS6nOg7xohwy4UlqUaUvw6wvHzZxuNnV8YWkz8CfVc9cEJsvmdII1+doe1QYkGxyhiI5Bf2cKgNJYxr539DwMKDMY2b0HpUFCTJYXpcYYJ8RhBroxsGGtt9pyPu0KZf6lScVPtyQMBzy0bODHRjkkS4q3F4xWEeBaMnh7BI6Vcta/uifB/WsoGRnRnjlZLQOhRgMSOq+nxINKpqDMlazvlk= 2021-05-14T15:30:42.973 INFO:teuthology.kill:2021-05-14 15:30:42,973.973 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-14T15:30:42.976 INFO:teuthology.kill:2021-05-14 15:30:42,976.976 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-14T15:30:43.029 INFO:teuthology.kill:2021-05-14 15:30:43,029.029 INFO:teuthology.orchestra.console:Power off smithi154 2021-05-14T15:30:43.040 INFO:teuthology.kill:2021-05-14 15:30:43,040.040 INFO:teuthology.orchestra.console:Power off smithi116 2021-05-14T15:30:43.167 INFO:teuthology.kill:2021-05-14 15:30:43,167.167 INFO:teuthology.orchestra.console:Power off for smithi154 completed 2021-05-14T15:30:43.177 INFO:teuthology.kill:2021-05-14 15:30:43,177.177 INFO:teuthology.orchestra.console:Power off for smithi116 completed 2021-05-14T15:30:43.305 INFO:teuthology.kill:2021-05-14 15:30:43,304.304 INFO:teuthology.lock.ops:unlocked smithi154.front.sepia.ceph.com 2021-05-14T15:30:43.315 INFO:teuthology.kill:2021-05-14 15:30:43,315.315 INFO:teuthology.lock.ops:unlocked smithi116.front.sepia.ceph.com 2021-05-14T15:32:43.523 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15