2021-05-28T05:24:48.706 INFO:root:teuthology version: 1.1.0-f359b10d
2021-05-28T05:24:48.813 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi196.front.sepia.ceph.com' reimaging
2021-05-28T05:24:48.814 INFO:teuthology.lock.ops:Updating [ubuntu@smithi196.front.sepia.ceph.com]: reset os type and version on server
2021-05-28T05:24:48.814 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server
2021-05-28T05:24:48.832 INFO:teuthology.lock.ops:Node 'ubuntu@smithi196.front.sepia.ceph.com' reimaging is complete
2021-05-28T05:24:48.833 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi204.front.sepia.ceph.com' reimaging
2021-05-28T05:24:48.833 INFO:teuthology.lock.ops:Updating [ubuntu@smithi204.front.sepia.ceph.com]: reset os type and version on server
2021-05-28T05:24:48.833 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server
2021-05-28T05:24:48.852 INFO:teuthology.lock.ops:Node 'ubuntu@smithi204.front.sepia.ceph.com' reimaging is complete
2021-05-28T05:24:49.038 INFO:teuthology.provision.fog.smithi196:Scheduling deploy of ubuntu 20.04
2021-05-28T05:24:49.049 INFO:teuthology.provision.fog.smithi204:Scheduling deploy of ubuntu 20.04
2021-05-28T05:24:49.393 INFO:teuthology.orchestra.console:Power off smithi204
2021-05-28T05:24:49.428 INFO:teuthology.orchestra.console:Power off smithi196
2021-05-28T05:24:57.828 INFO:teuthology.orchestra.console:Power off for smithi204 completed
2021-05-28T05:24:57.839 INFO:teuthology.orchestra.console:Power off for smithi196 completed
2021-05-28T05:24:57.929 INFO:teuthology.orchestra.console:Power on smithi204
2021-05-28T05:24:57.940 INFO:teuthology.orchestra.console:Power on smithi196
2021-05-28T05:25:06.342 INFO:teuthology.orchestra.console:Power on for smithi204 completed
2021-05-28T05:25:06.354 INFO:teuthology.orchestra.console:Power on for smithi196 completed
2021-05-28T05:25:06.444 INFO:teuthology.provision.fog.smithi204:Waiting for deploy to finish
2021-05-28T05:25:06.455 INFO:teuthology.provision.fog.smithi196:Waiting for deploy to finish
2021-05-28T05:30:05.885 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-05-28T05:30:06.418 INFO:teuthology.provision.fog.smithi196:Node is ready
2021-05-28T05:30:06.425 INFO:teuthology.orchestra.run.smithi196.stdout:smithi196.front.sepia.ceph.com
2021-05-28T05:30:06.476 INFO:teuthology.orchestra.run.smithi196.stdout:172.21.15.196 smithi196.front.sepia.ceph.com smithi196
2021-05-28T05:30:06.651 INFO:teuthology.provision.fog.smithi196:Deploy complete!
2021-05-28T05:30:26.127 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-05-28T05:30:26.603 INFO:teuthology.provision.fog.smithi204:Node is ready
2021-05-28T05:30:26.615 INFO:teuthology.orchestra.run.smithi204.stdout:smithi204.front.sepia.ceph.com
2021-05-28T05:30:26.664 INFO:teuthology.orchestra.run.smithi204.stdout:172.21.15.204 smithi204.front.sepia.ceph.com smithi204
2021-05-28T05:30:26.864 INFO:teuthology.provision.fog.smithi204:Deploy complete!
2021-05-28T05:30:27.018 INFO:teuthology.lock.ops:Checking smithi196.front.sepia.ceph.com
2021-05-28T05:30:27.019 INFO:teuthology.lock.ops:New key found. Updating...
2021-05-28T05:30:27.034 INFO:teuthology.lock.ops:Checking smithi204.front.sepia.ceph.com
2021-05-28T05:30:27.035 INFO:teuthology.lock.ops:New key found. Updating...
2021-05-28T05:30:27.049 INFO:teuthology.lock.ops:Updating [smithi196.front.sepia.ceph.com]: set os type and version on server
2021-05-28T05:30:27.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-05-28T05:30:27.241 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64
2021-05-28T05:30:27.290 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="Ubuntu"
2021-05-28T05:30:27.291 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="20.04.2 LTS (Focal Fossa)"
2021-05-28T05:30:27.291 INFO:teuthology.orchestra.run.smithi196.stdout:ID=ubuntu
2021-05-28T05:30:27.291 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE=debian
2021-05-28T05:30:27.291 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS"
2021-05-28T05:30:27.292 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="20.04"
2021-05-28T05:30:27.292 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://www.ubuntu.com/"
2021-05-28T05:30:27.292 INFO:teuthology.orchestra.run.smithi196.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-05-28T05:30:27.292 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-05-28T05:30:27.292 INFO:teuthology.orchestra.run.smithi196.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-05-28T05:30:27.293 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_CODENAME=focal
2021-05-28T05:30:27.293 INFO:teuthology.orchestra.run.smithi196.stdout:UBUNTU_CODENAME=focal
2021-05-28T05:30:27.294 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server
2021-05-28T05:30:27.308 INFO:teuthology.lock.ops:Updating [smithi204.front.sepia.ceph.com]: set os type and version on server
2021-05-28T05:30:27.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-05-28T05:30:27.470 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64
2021-05-28T05:30:27.475 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="Ubuntu"
2021-05-28T05:30:27.475 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="20.04.2 LTS (Focal Fossa)"
2021-05-28T05:30:27.476 INFO:teuthology.orchestra.run.smithi204.stdout:ID=ubuntu
2021-05-28T05:30:27.476 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE=debian
2021-05-28T05:30:27.476 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS"
2021-05-28T05:30:27.476 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="20.04"
2021-05-28T05:30:27.477 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://www.ubuntu.com/"
2021-05-28T05:30:27.477 INFO:teuthology.orchestra.run.smithi204.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-05-28T05:30:27.477 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-05-28T05:30:27.477 INFO:teuthology.orchestra.run.smithi204.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-05-28T05:30:27.477 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_CODENAME=focal
2021-05-28T05:30:27.478 INFO:teuthology.orchestra.run.smithi204.stdout:UBUNTU_CODENAME=focal
2021-05-28T05:30:27.478 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server
2021-05-28T05:30:27.559 INFO:teuthology.dispatcher.supervisor:Running job 6140296
2021-05-28T05:30:27.559 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology -v --owner scheduled_xiubli@teuthology --archive /home/teuthworker/archive/xiubli-2021-05-28_05:23:05-fs:functional:-wip-lxb-20210527-45434-pg-num3-distro-basic-smithi/6140296 --name xiubli-2021-05-28_05:23:05-fs:functional:-wip-lxb-20210527-45434-pg-num3-distro-basic-smithi --description fs:functional:/{begin.yaml clusters/1a3s-mds-4c-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} distro/{rhel_8.yaml} mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore/bluestore-ec-root.yaml overrides/{frag_enable.yaml no_client_pidfile.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/mds-full.yaml} -- /home/teuthworker/archive/xiubli-2021-05-28_05:23:05-fs:functional:-wip-lxb-20210527-45434-pg-num3-distro-basic-smithi/6140296/orig.config.yaml
2021-05-28T05:30:27.564 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/xiubli-2021-05-28_05:23:05-fs:functional:-wip-lxb-20210527-45434-pg-num3-distro-basic-smithi/6140296
2021-05-28T05:30:27.564 INFO:teuthology.dispatcher.supervisor:Job PID: 15706
2021-05-28T05:30:27.565 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-05-28T17:31:16.551 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-05-28T17:31:16.709 INFO:teuthology.kill:Killing Pids: {15706}
2021-05-28T17:31:17.639 INFO:teuthology.kill:Nuking machines: ['smithi196', 'smithi204']
2021-05-28T17:31:18.456 INFO:teuthology.kill:2021-05-28 17:31:18,455.455 INFO:teuthology.nuke:targets:
2021-05-28T17:31:18.457 INFO:teuthology.kill:  smithi196.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrZiJaFwn0cikrMFq+s/wps+i0Ph0xmC+3WbptO2P7Xf4I0302BoRUzkEqp3tCjQ2F+VtyHsitNCqqcv4Faxpqe2s6rilo5fakDc01tXBF228ggVZpVxN2w3hXR0iy0ZMCsyecgq9SNH9A7lsj1ImX3vhMDke17nxolrPtkuCQvJUZ6vl24cacDcGa4PfVsv44jY0RcT9fH2PdlGAMFIaTT6M2ITf6Cwh5uVtAarGmf3I0pH1ZsfQm0+whn4rvUoSUoolOaRH7PEuWKEwLEvf7gNr/cD2s06dyKmK3870ocx3QaHbRI43jeKmSymr2StA+LVKTsAdXvnSp7EOSwljauTyCTplk3vmb9WYQ/gWSGQr+P6HaH2fsS1Nqw4+qRbwumAedT9msAEeCBvxw8IveikfrbSimHM+TiYQFfYvo/baf2DGR5MqYUjpeN68oBaFeYSbD5R3StaZmBGckaKpPmPg3vxVm3W7UHFGo1kollMTAYaA8EdIzD4jlbjefF1E=
2021-05-28T17:31:18.457 INFO:teuthology.kill:  smithi204.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXHHget3FERmgDt/5KcfmTFeIPJxAlcaA+jqABs80tUu5NDdli39oIjBfv4ngFvjC/r0miDhLYTKsDBc+cOwXSiPZUIjFN8qvUV3PLCZ/cKJqaPLoTft2hGsnnzoN0CcAStUeXT5XwbLAhjmElJ8dn3xVpKEdjpxUctNioRt0UmbQngGGXamOrmoup2T1dsjil8vK8CSYyzP3zXqInUMevSdjAH95NPNXddE0qSImiAuO8AqR7yE3XTiqCMrJqwQODyJn5JLqgKZJ5Icd1aNOwjUC6421jypQ3mUaz3TAbLRCaBoZakRTjqpoUeVOT7IkKIs5cBf5+TqnIYbwlrV8mMaTTNSelDjZsVgYC5SaFjDuFo9TecQVcdgocJRsG4Q8+JuF9gKVT31PA6WXGEonBvXZ47WOqWemFRqz0Uf73r77VGk/cqNMHxi/bWaaYVhu+B9FEm1Icp1DVVqxpsPOWkq1lXzM6ooBv9ZpyL0WBn0T8A8FN6JF/n6iIAsY8tXU=
2021-05-28T17:31:18.457 INFO:teuthology.kill:2021-05-28 17:31:18,456.456 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-28T17:31:18.458 INFO:teuthology.kill:2021-05-28 17:31:18,458.458 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-28T17:31:18.513 INFO:teuthology.kill:2021-05-28 17:31:18,513.513 INFO:teuthology.orchestra.console:Power off smithi204
2021-05-28T17:31:18.525 INFO:teuthology.kill:2021-05-28 17:31:18,525.525 INFO:teuthology.orchestra.console:Power off smithi196
2021-05-28T17:31:30.994 INFO:teuthology.kill:2021-05-28 17:31:30,994.994 INFO:teuthology.orchestra.console:Power off for smithi204 completed
2021-05-28T17:31:31.000 INFO:teuthology.kill:2021-05-28 17:31:31,000.000 INFO:teuthology.orchestra.console:Power off for smithi196 completed
2021-05-28T17:31:31.208 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2']
2021-05-28T17:31:31.208 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']
2021-05-28T17:31:31.209 INFO:teuthology.misc:Compressing logs...
2021-05-28T17:31:31.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-05-28T17:32:31.237 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in <listcomp>
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 434, in run
    transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-05-28T17:32:31.300 INFO:teuthology.kill:No teuthology processes running
2021-05-28T17:32:32.246 INFO:teuthology.kill:Nuking machines: ['smithi196', 'smithi204']
2021-05-28T17:32:33.132 INFO:teuthology.kill:2021-05-28 17:32:33,132.132 INFO:teuthology.nuke:targets:
2021-05-28T17:32:33.133 INFO:teuthology.kill:  smithi196.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDrZiJaFwn0cikrMFq+s/wps+i0Ph0xmC+3WbptO2P7Xf4I0302BoRUzkEqp3tCjQ2F+VtyHsitNCqqcv4Faxpqe2s6rilo5fakDc01tXBF228ggVZpVxN2w3hXR0iy0ZMCsyecgq9SNH9A7lsj1ImX3vhMDke17nxolrPtkuCQvJUZ6vl24cacDcGa4PfVsv44jY0RcT9fH2PdlGAMFIaTT6M2ITf6Cwh5uVtAarGmf3I0pH1ZsfQm0+whn4rvUoSUoolOaRH7PEuWKEwLEvf7gNr/cD2s06dyKmK3870ocx3QaHbRI43jeKmSymr2StA+LVKTsAdXvnSp7EOSwljauTyCTplk3vmb9WYQ/gWSGQr+P6HaH2fsS1Nqw4+qRbwumAedT9msAEeCBvxw8IveikfrbSimHM+TiYQFfYvo/baf2DGR5MqYUjpeN68oBaFeYSbD5R3StaZmBGckaKpPmPg3vxVm3W7UHFGo1kollMTAYaA8EdIzD4jlbjefF1E=
2021-05-28T17:32:33.134 INFO:teuthology.kill:  smithi204.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXHHget3FERmgDt/5KcfmTFeIPJxAlcaA+jqABs80tUu5NDdli39oIjBfv4ngFvjC/r0miDhLYTKsDBc+cOwXSiPZUIjFN8qvUV3PLCZ/cKJqaPLoTft2hGsnnzoN0CcAStUeXT5XwbLAhjmElJ8dn3xVpKEdjpxUctNioRt0UmbQngGGXamOrmoup2T1dsjil8vK8CSYyzP3zXqInUMevSdjAH95NPNXddE0qSImiAuO8AqR7yE3XTiqCMrJqwQODyJn5JLqgKZJ5Icd1aNOwjUC6421jypQ3mUaz3TAbLRCaBoZakRTjqpoUeVOT7IkKIs5cBf5+TqnIYbwlrV8mMaTTNSelDjZsVgYC5SaFjDuFo9TecQVcdgocJRsG4Q8+JuF9gKVT31PA6WXGEonBvXZ47WOqWemFRqz0Uf73r77VGk/cqNMHxi/bWaaYVhu+B9FEm1Icp1DVVqxpsPOWkq1lXzM6ooBv9ZpyL0WBn0T8A8FN6JF/n6iIAsY8tXU=
2021-05-28T17:32:33.152 INFO:teuthology.kill:2021-05-28 17:32:33,152.152 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-28T17:32:33.155 INFO:teuthology.kill:2021-05-28 17:32:33,155.155 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-28T17:32:33.209 INFO:teuthology.kill:2021-05-28 17:32:33,208.208 INFO:teuthology.orchestra.console:Power off smithi196
2021-05-28T17:32:33.221 INFO:teuthology.kill:2021-05-28 17:32:33,221.221 INFO:teuthology.orchestra.console:Power off smithi204
2021-05-28T17:32:33.347 INFO:teuthology.kill:2021-05-28 17:32:33,347.347 INFO:teuthology.orchestra.console:Power off for smithi196 completed
2021-05-28T17:32:33.357 INFO:teuthology.kill:2021-05-28 17:32:33,357.357 INFO:teuthology.orchestra.console:Power off for smithi204 completed
2021-05-28T17:32:33.492 INFO:teuthology.kill:2021-05-28 17:32:33,492.492 INFO:teuthology.lock.ops:unlocked smithi196.front.sepia.ceph.com
2021-05-28T17:32:33.494 INFO:teuthology.kill:2021-05-28 17:32:33,493.493 ERROR:teuthology.lock.ops:failed to unlock smithi204.front.sepia.ceph.com. reason: 500
2021-05-28T17:32:33.576 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/xiubli-2021-05-28_05:23:05-fs:functional:-wip-lxb-20210527-45434-pg-num3-distro-basic-smithi/jobs/ failed with status 500: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>
    
  </body>
</html>

2021-05-28T17:34:34.654 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-05-28T17:34:34.748 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2021-05-28T17:34:34.748 INFO:teuthology.nuke:Checking targets against current locks
2021-05-28T17:34:34.813 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology-dispatcher", line 33, in <module>
    sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')())
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/scripts/dispatcher.py", line 33, in main
    teuthology.dispatcher.main(args)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/__init__.py", line 61, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 59, in main
    verbose
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 152, in run_job
    unlock_targets(job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 196, in unlock_targets
    nuke(fake_ctx, True)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/nuke/__init__.py", line 250, in nuke
    if ctx.name not in lock['description']:
TypeError: argument of type 'NoneType' is not iterable