2021-08-17T09:41:32.468 INFO:root:teuthology version: 1.1.0-321319b1 2021-08-17T09:41:32.569 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi006.front.sepia.ceph.com' reimaging 2021-08-17T09:41:32.571 INFO:teuthology.lock.ops:Updating [ubuntu@smithi006.front.sepia.ceph.com]: reset os type and version on server 2021-08-17T09:41:32.571 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2021-08-17T09:41:32.586 INFO:teuthology.lock.ops:Node 'ubuntu@smithi006.front.sepia.ceph.com' reimaging is complete 2021-08-17T09:41:32.586 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi120.front.sepia.ceph.com' reimaging 2021-08-17T09:41:32.587 INFO:teuthology.lock.ops:Updating [ubuntu@smithi120.front.sepia.ceph.com]: reset os type and version on server 2021-08-17T09:41:32.587 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2021-08-17T09:41:32.602 INFO:teuthology.lock.ops:Node 'ubuntu@smithi120.front.sepia.ceph.com' reimaging is complete 2021-08-17T09:41:32.797 INFO:teuthology.provision.fog.smithi006:Scheduling deploy of ubuntu 20.04 2021-08-17T09:41:32.809 INFO:teuthology.provision.fog.smithi120:Scheduling deploy of ubuntu 20.04 2021-08-17T09:41:33.132 INFO:teuthology.orchestra.console:Power off smithi006 2021-08-17T09:41:33.148 INFO:teuthology.orchestra.console:Power off smithi120 2021-08-17T09:41:41.512 INFO:teuthology.orchestra.console:Power off for smithi120 completed 2021-08-17T09:41:41.614 INFO:teuthology.orchestra.console:Power on smithi120 2021-08-17T09:41:45.616 INFO:teuthology.orchestra.console:Power off for smithi006 completed 2021-08-17T09:41:45.722 INFO:teuthology.orchestra.console:Power on smithi006 2021-08-17T09:41:49.983 INFO:teuthology.orchestra.console:Power on for smithi120 completed 2021-08-17T09:41:50.084 INFO:teuthology.provision.fog.smithi120:Waiting for deploy to finish 2021-08-17T09:41:54.090 INFO:teuthology.orchestra.console:Power on for smithi006 completed 2021-08-17T09:41:54.192 INFO:teuthology.provision.fog.smithi006:Waiting for deploy to finish 2021-08-17T09:46:31.845 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-17T09:46:32.394 INFO:teuthology.provision.fog.smithi006:Node is ready 2021-08-17T09:46:32.401 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006.front.sepia.ceph.com 2021-08-17T09:46:32.407 INFO:teuthology.orchestra.run.smithi006.stdout:172.21.15.6 smithi006.front.sepia.ceph.com smithi006 2021-08-17T09:46:32.701 INFO:teuthology.provision.fog.smithi006:Deploy complete! 2021-08-17T09:46:39.597 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-17T09:46:40.075 INFO:teuthology.provision.fog.smithi120:Node is ready 2021-08-17T09:46:40.082 INFO:teuthology.orchestra.run.smithi120.stdout:smithi120.front.sepia.ceph.com 2021-08-17T09:46:40.089 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.15.120 smithi120.front.sepia.ceph.com smithi120 2021-08-17T09:46:40.285 INFO:teuthology.provision.fog.smithi120:Deploy complete! 2021-08-17T09:46:40.457 INFO:teuthology.lock.ops:Checking smithi006.front.sepia.ceph.com 2021-08-17T09:46:40.459 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-17T09:46:40.476 INFO:teuthology.lock.ops:Checking smithi120.front.sepia.ceph.com 2021-08-17T09:46:40.476 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-17T09:46:40.490 INFO:teuthology.lock.ops:Updating [smithi006.front.sepia.ceph.com]: set os type and version on server 2021-08-17T09:46:40.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-17T09:46:40.674 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2021-08-17T09:46:40.681 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="Ubuntu" 2021-08-17T09:46:40.681 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-08-17T09:46:40.682 INFO:teuthology.orchestra.run.smithi006.stdout:ID=ubuntu 2021-08-17T09:46:40.682 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE=debian 2021-08-17T09:46:40.682 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-08-17T09:46:40.682 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="20.04" 2021-08-17T09:46:40.683 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://www.ubuntu.com/" 2021-08-17T09:46:40.683 INFO:teuthology.orchestra.run.smithi006.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-08-17T09:46:40.683 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-08-17T09:46:40.683 INFO:teuthology.orchestra.run.smithi006.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-08-17T09:46:40.684 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_CODENAME=focal 2021-08-17T09:46:40.684 INFO:teuthology.orchestra.run.smithi006.stdout:UBUNTU_CODENAME=focal 2021-08-17T09:46:40.685 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2021-08-17T09:46:40.707 INFO:teuthology.lock.ops:Updating [smithi120.front.sepia.ceph.com]: set os type and version on server 2021-08-17T09:46:40.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-17T09:46:40.887 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64 2021-08-17T09:46:40.938 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="Ubuntu" 2021-08-17T09:46:40.939 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-08-17T09:46:40.939 INFO:teuthology.orchestra.run.smithi120.stdout:ID=ubuntu 2021-08-17T09:46:40.939 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE=debian 2021-08-17T09:46:40.940 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-08-17T09:46:40.940 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="20.04" 2021-08-17T09:46:40.940 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://www.ubuntu.com/" 2021-08-17T09:46:40.940 INFO:teuthology.orchestra.run.smithi120.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-08-17T09:46:40.940 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-08-17T09:46:40.941 INFO:teuthology.orchestra.run.smithi120.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-08-17T09:46:40.941 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_CODENAME=focal 2021-08-17T09:46:40.941 INFO:teuthology.orchestra.run.smithi120.stdout:UBUNTU_CODENAME=focal 2021-08-17T09:46:40.942 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2021-08-17T09:46:41.044 INFO:teuthology.dispatcher.supervisor:Running job 6341834 2021-08-17T09:46:41.045 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-08-16_19:26:09-fs-wip-yuri3-testing-2021-08-16-0913-distro-basic-smithi/6341834 --name yuriw-2021-08-16_19:26:09-fs-wip-yuri3-testing-2021-08-16-0913-distro-basic-smithi --description fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/fragment} -- /home/teuthworker/archive/yuriw-2021-08-16_19:26:09-fs-wip-yuri3-testing-2021-08-16-0913-distro-basic-smithi/6341834/orig.config.yaml 2021-08-17T09:46:41.049 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-08-16_19:26:09-fs-wip-yuri3-testing-2021-08-16-0913-distro-basic-smithi/6341834 2021-08-17T09:46:41.050 INFO:teuthology.dispatcher.supervisor:Job PID: 23477 2021-08-17T09:46:41.051 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-08-17T21:47:32.851 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-08-17T21:47:33.091 INFO:teuthology.kill:Killing Pids: {23477} 2021-08-17T21:47:34.230 INFO:teuthology.kill:Nuking machines: ['smithi006', 'smithi120'] 2021-08-17T21:47:35.231 INFO:teuthology.kill:2021-08-17 21:47:35,230.230 INFO:teuthology.nuke:targets: 2021-08-17T21:47:35.231 INFO:teuthology.kill: smithi006.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0n+xaKhWyqjxyTlfUUKjkjkiRxlknwPuveQmrQziJffFNUNBei4F/E6i9yELAYgsITdst9tGB4wfe18YaUvQC/kOO2jmzLTmHoNquzdoYXHSG4a3zcgi8CQnc1HBejCRXlAr4izGroqgGAmzaWJcodlok4ACsBtIEamCP1SIQjgzgBRsdPJPJfDOszt8siS9fcbNvjui0QA3M8/0/qSg+XCeZdU8smewVCl4b1g2xqKFzi+gBfeg7CBmRzt8l3MuCteHPmu7nbE5c0ZB4FilHeKqktmxH0DJ1STHDiEusZwWoFeUbvmeXeG9CdE7+xNM35/C0Mmnl/8R01G2XJ8G0B0XkuNPujoQwm5qN2+PLKitlaqERpPHtO0fULbpdhZrqVUgUFYf9bTOkjQ4TdFkxXwnZk8FF5GShgz8ffYqqvs0wnJ1DvTCE+bcIW9foRnJ8Dc9sjJCqBryJ3GTc8nNcwntxGpX7NAvw5pNWZK/OgEjVVjJXS1Py+RRD6wjzUFs= 2021-08-17T21:47:35.232 INFO:teuthology.kill: smithi120.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDaDh3o52jMhq1KZZEohL2lr26SgJTgMgMbZr+/j2+mdEBOi0PkANbTWaIj3Qk3yVOklZvhGmrUoaugnI+OPa07A4QBp1yCHsrjJWJTcI+dNtZpLfLjOgtGb1rLE97A0H5tQqqBibKQfVV4Ol0WIOZGdkyJJtF5+Z71wvrvAAB2FdsLNo24Hg5BWZO9cQk+jnLx/7rV4Zb3UuxWeTcg9PCKuPXGFrSl6UTBvbjyO7DHFxrWzLrD4van6M+ivrSX1foA/C4WmTTLCtwx8OR2+E9KO+A8YOXfK5JyPTli5ie6pNM7STtpUQ4wDbvplBS30sR3Jh8bJ05daawU7yVX29rF9hkYZ/tgbBgqQaCfhaEAmwB8KlI9OcmXwHT1JmDvEhJ+RH/AwWeaFqo6FtYX/GfVOuyrBsFWwU0jXcvWQb2bS80wmkZx9cxlI6AwAjF9NiypSx38nqnbZkoZBRxkb5CUIZ4wNiXxitjqclO/HEZ56I56kTyQsBJTLXN8lvOema0= 2021-08-17T21:47:35.232 INFO:teuthology.kill:2021-08-17 21:47:35,231.231 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-17T21:47:35.233 INFO:teuthology.kill:2021-08-17 21:47:35,233.233 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-17T21:47:35.300 INFO:teuthology.kill:2021-08-17 21:47:35,299.299 INFO:teuthology.orchestra.console:Power off smithi006 2021-08-17T21:47:35.334 INFO:teuthology.kill:2021-08-17 21:47:35,333.333 INFO:teuthology.orchestra.console:Power off smithi120 2021-08-17T21:47:43.673 INFO:teuthology.kill:2021-08-17 21:47:43,672.672 INFO:teuthology.orchestra.console:Power off for smithi006 completed 2021-08-17T21:47:43.691 INFO:teuthology.kill:2021-08-17 21:47:43,691.691 INFO:teuthology.orchestra.console:Power off for smithi120 completed 2021-08-17T21:47:43.903 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2021-08-17T21:47:43.904 INFO:teuthology.task.internal:roles: ubuntu@smithi120.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2021-08-17T21:47:43.904 INFO:teuthology.misc:Compressing logs... 2021-08-17T21:47:43.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-17T21:48:43.969 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-08-17T21:48:44.034 INFO:teuthology.kill:No teuthology processes running 2021-08-17T21:48:45.207 INFO:teuthology.kill:Nuking machines: ['smithi006', 'smithi120'] 2021-08-17T21:48:46.198 INFO:teuthology.kill:2021-08-17 21:48:46,197.197 INFO:teuthology.nuke:targets: 2021-08-17T21:48:46.199 INFO:teuthology.kill: smithi006.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0n+xaKhWyqjxyTlfUUKjkjkiRxlknwPuveQmrQziJffFNUNBei4F/E6i9yELAYgsITdst9tGB4wfe18YaUvQC/kOO2jmzLTmHoNquzdoYXHSG4a3zcgi8CQnc1HBejCRXlAr4izGroqgGAmzaWJcodlok4ACsBtIEamCP1SIQjgzgBRsdPJPJfDOszt8siS9fcbNvjui0QA3M8/0/qSg+XCeZdU8smewVCl4b1g2xqKFzi+gBfeg7CBmRzt8l3MuCteHPmu7nbE5c0ZB4FilHeKqktmxH0DJ1STHDiEusZwWoFeUbvmeXeG9CdE7+xNM35/C0Mmnl/8R01G2XJ8G0B0XkuNPujoQwm5qN2+PLKitlaqERpPHtO0fULbpdhZrqVUgUFYf9bTOkjQ4TdFkxXwnZk8FF5GShgz8ffYqqvs0wnJ1DvTCE+bcIW9foRnJ8Dc9sjJCqBryJ3GTc8nNcwntxGpX7NAvw5pNWZK/OgEjVVjJXS1Py+RRD6wjzUFs= 2021-08-17T21:48:46.199 INFO:teuthology.kill: smithi120.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDaDh3o52jMhq1KZZEohL2lr26SgJTgMgMbZr+/j2+mdEBOi0PkANbTWaIj3Qk3yVOklZvhGmrUoaugnI+OPa07A4QBp1yCHsrjJWJTcI+dNtZpLfLjOgtGb1rLE97A0H5tQqqBibKQfVV4Ol0WIOZGdkyJJtF5+Z71wvrvAAB2FdsLNo24Hg5BWZO9cQk+jnLx/7rV4Zb3UuxWeTcg9PCKuPXGFrSl6UTBvbjyO7DHFxrWzLrD4van6M+ivrSX1foA/C4WmTTLCtwx8OR2+E9KO+A8YOXfK5JyPTli5ie6pNM7STtpUQ4wDbvplBS30sR3Jh8bJ05daawU7yVX29rF9hkYZ/tgbBgqQaCfhaEAmwB8KlI9OcmXwHT1JmDvEhJ+RH/AwWeaFqo6FtYX/GfVOuyrBsFWwU0jXcvWQb2bS80wmkZx9cxlI6AwAjF9NiypSx38nqnbZkoZBRxkb5CUIZ4wNiXxitjqclO/HEZ56I56kTyQsBJTLXN8lvOema0= 2021-08-17T21:48:46.223 INFO:teuthology.kill:2021-08-17 21:48:46,223.223 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-17T21:48:46.231 INFO:teuthology.kill:2021-08-17 21:48:46,231.231 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-17T21:48:46.290 INFO:teuthology.kill:2021-08-17 21:48:46,290.290 INFO:teuthology.orchestra.console:Power off smithi120 2021-08-17T21:48:46.303 INFO:teuthology.kill:2021-08-17 21:48:46,303.303 INFO:teuthology.orchestra.console:Power off smithi006 2021-08-17T21:48:46.430 INFO:teuthology.kill:2021-08-17 21:48:46,430.430 INFO:teuthology.orchestra.console:Power off for smithi120 completed 2021-08-17T21:48:46.443 INFO:teuthology.kill:2021-08-17 21:48:46,442.442 INFO:teuthology.orchestra.console:Power off for smithi006 completed 2021-08-17T21:48:46.576 INFO:teuthology.kill:2021-08-17 21:48:46,575.575 INFO:teuthology.lock.ops:unlocked smithi120.front.sepia.ceph.com 2021-08-17T21:48:46.580 INFO:teuthology.kill:2021-08-17 21:48:46,579.579 INFO:teuthology.lock.ops:unlocked smithi006.front.sepia.ceph.com 2021-08-17T21:50:46.865 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-08-17T21:50:46.972 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-08-17T21:50:46.972 INFO:teuthology.nuke:Checking targets against current locks 2021-08-17T21:50:47.099 INFO:teuthology.nuke:Not nuking smithi006.front.sepia.ceph.com because description doesn't match 2021-08-17T21:50:47.100 INFO:teuthology.nuke:Not nuking smithi120.front.sepia.ceph.com because description doesn't match