2021-05-26T03:57:31.990 INFO:root:teuthology version: 1.1.0-0c70841a 2021-05-26T03:57:32.084 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi137.front.sepia.ceph.com' reimaging 2021-05-26T03:57:32.085 INFO:teuthology.lock.ops:Updating [ubuntu@smithi137.front.sepia.ceph.com]: reset os type and version on server 2021-05-26T03:57:32.085 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2021-05-26T03:57:32.099 INFO:teuthology.lock.ops:Node 'ubuntu@smithi137.front.sepia.ceph.com' reimaging is complete 2021-05-26T03:57:32.100 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi205.front.sepia.ceph.com' reimaging 2021-05-26T03:57:32.100 INFO:teuthology.lock.ops:Updating [ubuntu@smithi205.front.sepia.ceph.com]: reset os type and version on server 2021-05-26T03:57:32.100 INFO:teuthology.lock.ops:Updating smithi205.front.sepia.ceph.com on lock server 2021-05-26T03:57:32.115 INFO:teuthology.lock.ops:Node 'ubuntu@smithi205.front.sepia.ceph.com' reimaging is complete 2021-05-26T03:57:32.339 INFO:teuthology.provision.fog.smithi137:Scheduling deploy of ubuntu 20.04 2021-05-26T03:57:32.440 INFO:teuthology.provision.fog.smithi205:Scheduling deploy of ubuntu 20.04 2021-05-26T03:57:32.764 INFO:teuthology.orchestra.console:Power off smithi137 2021-05-26T03:57:32.858 INFO:teuthology.orchestra.console:Power off smithi205 2021-05-26T03:57:41.138 INFO:teuthology.orchestra.console:Power off for smithi137 completed 2021-05-26T03:57:41.225 INFO:teuthology.orchestra.console:Power off for smithi205 completed 2021-05-26T03:57:41.240 INFO:teuthology.orchestra.console:Power on smithi137 2021-05-26T03:57:41.327 INFO:teuthology.orchestra.console:Power on smithi205 2021-05-26T03:57:49.605 INFO:teuthology.orchestra.console:Power on for smithi137 completed 2021-05-26T03:57:49.707 INFO:teuthology.provision.fog.smithi137:Waiting for deploy to finish 2021-05-26T03:57:49.776 INFO:teuthology.orchestra.console:Power on for smithi205 completed 2021-05-26T03:57:49.877 INFO:teuthology.provision.fog.smithi205:Waiting for deploy to finish 2021-05-26T04:02:42.734 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-26T04:02:43.176 INFO:teuthology.provision.fog.smithi137:Node is ready 2021-05-26T04:02:43.184 INFO:teuthology.orchestra.run.smithi137.stdout:smithi137.front.sepia.ceph.com 2021-05-26T04:02:43.192 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.15.137 smithi137.front.sepia.ceph.com smithi137 2021-05-26T04:02:43.436 INFO:teuthology.provision.fog.smithi137:Deploy complete! 2021-05-26T04:02:47.344 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-26T04:02:47.822 INFO:teuthology.provision.fog.smithi205:Node is ready 2021-05-26T04:02:47.871 INFO:teuthology.orchestra.run.smithi205.stdout:smithi205.front.sepia.ceph.com 2021-05-26T04:02:47.878 INFO:teuthology.orchestra.run.smithi205.stdout:172.21.15.205 smithi205.front.sepia.ceph.com smithi205 2021-05-26T04:02:48.074 INFO:teuthology.provision.fog.smithi205:Deploy complete! 2021-05-26T04:02:48.225 INFO:teuthology.lock.ops:Checking smithi137.front.sepia.ceph.com 2021-05-26T04:02:48.226 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-26T04:02:48.241 INFO:teuthology.lock.ops:Checking smithi205.front.sepia.ceph.com 2021-05-26T04:02:48.241 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-26T04:02:48.255 INFO:teuthology.lock.ops:Updating [smithi137.front.sepia.ceph.com]: set os type and version on server 2021-05-26T04:02:48.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-26T04:02:48.426 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2021-05-26T04:02:48.432 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="Ubuntu" 2021-05-26T04:02:48.433 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-05-26T04:02:48.433 INFO:teuthology.orchestra.run.smithi137.stdout:ID=ubuntu 2021-05-26T04:02:48.433 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE=debian 2021-05-26T04:02:48.434 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-05-26T04:02:48.434 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="20.04" 2021-05-26T04:02:48.434 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://www.ubuntu.com/" 2021-05-26T04:02:48.434 INFO:teuthology.orchestra.run.smithi137.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-05-26T04:02:48.434 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-05-26T04:02:48.435 INFO:teuthology.orchestra.run.smithi137.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-05-26T04:02:48.435 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_CODENAME=focal 2021-05-26T04:02:48.435 INFO:teuthology.orchestra.run.smithi137.stdout:UBUNTU_CODENAME=focal 2021-05-26T04:02:48.436 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2021-05-26T04:02:48.450 INFO:teuthology.lock.ops:Updating [smithi205.front.sepia.ceph.com]: set os type and version on server 2021-05-26T04:02:48.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-26T04:02:48.618 INFO:teuthology.orchestra.run.smithi205.stdout:x86_64 2021-05-26T04:02:48.625 INFO:teuthology.orchestra.run.smithi205.stdout:NAME="Ubuntu" 2021-05-26T04:02:48.625 INFO:teuthology.orchestra.run.smithi205.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-05-26T04:02:48.625 INFO:teuthology.orchestra.run.smithi205.stdout:ID=ubuntu 2021-05-26T04:02:48.625 INFO:teuthology.orchestra.run.smithi205.stdout:ID_LIKE=debian 2021-05-26T04:02:48.626 INFO:teuthology.orchestra.run.smithi205.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-05-26T04:02:48.626 INFO:teuthology.orchestra.run.smithi205.stdout:VERSION_ID="20.04" 2021-05-26T04:02:48.626 INFO:teuthology.orchestra.run.smithi205.stdout:HOME_URL="https://www.ubuntu.com/" 2021-05-26T04:02:48.626 INFO:teuthology.orchestra.run.smithi205.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-05-26T04:02:48.627 INFO:teuthology.orchestra.run.smithi205.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-05-26T04:02:48.627 INFO:teuthology.orchestra.run.smithi205.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-05-26T04:02:48.627 INFO:teuthology.orchestra.run.smithi205.stdout:VERSION_CODENAME=focal 2021-05-26T04:02:48.627 INFO:teuthology.orchestra.run.smithi205.stdout:UBUNTU_CODENAME=focal 2021-05-26T04:02:48.628 INFO:teuthology.lock.ops:Updating smithi205.front.sepia.ceph.com on lock server 2021-05-26T04:02:48.703 INFO:teuthology.dispatcher.supervisor:Running job 6134730 2021-05-26T04:02:48.703 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-05-25_19:30:22-fs-wip-yuri4-testing-2021-05-25-0940-pacific-distro-basic-smithi/6134730 --name yuriw-2021-05-25_19:30:22-fs-wip-yuri4-testing-2021-05-25-0940-pacific-distro-basic-smithi --description fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-full} -- /home/teuthworker/archive/yuriw-2021-05-25_19:30:22-fs-wip-yuri4-testing-2021-05-25-0940-pacific-distro-basic-smithi/6134730/orig.config.yaml 2021-05-26T04:02:48.707 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-05-25_19:30:22-fs-wip-yuri4-testing-2021-05-25-0940-pacific-distro-basic-smithi/6134730 2021-05-26T04:02:48.708 INFO:teuthology.dispatcher.supervisor:Job PID: 29373 2021-05-26T04:02:48.708 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-26T16:03:41.924 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-26T16:03:42.135 INFO:teuthology.kill:Killing Pids: {29373} 2021-05-26T16:03:43.233 INFO:teuthology.kill:Nuking machines: ['smithi137', 'smithi205'] 2021-05-26T16:03:44.170 INFO:teuthology.kill:2021-05-26 16:03:44,170.170 INFO:teuthology.nuke:targets: 2021-05-26T16:03:44.171 INFO:teuthology.kill: smithi137.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtlSkfeU5vTP+BuAtZdxvJxZ4RlaWR9oflAc5D5fSDJNrkvVEzuDBUsJVOTUJHZDBVCXUekqglJ5p8e3CnHhsQ4XjoY8Wa6dU16rCw8CohkSnfDf3HpDV3c2/gQDXv7Js6nZQnzm7Kw6OZNWaxjZIVX0GF67G7s7Xdlpz1+tUAK7GD6uVY0wKv3M6OXV3TSi4hgAaJbek3krbWAcfwxB6kTY9XL96J/u5jjs3e6iKv+5ltypjSZvEsnZndO7TFNMQbVTRlfzYXeh5DZm2Rn7CcAxH072ynX0wU/r92jSQYW/FltVoIfYXN+f6elG4q3MJZqApN7g/hbbpiCFBO1Wx0qbTOakNeLf/bbKIcMpBwJ5m8lffBWGDCz0XvyZWbzaaANjWNCOuz5NGORGYLMARRoZZyknh9d3ea63kqVVmPeFPlUl7cqkZv93k30PAgSHUMuxBLnLsxsZUEYr6Jz65tKbftXP5qBLvbFtEUzY1VCcVPL1Y+n5X3GggbGaT6C9s= 2021-05-26T16:03:44.171 INFO:teuthology.kill: smithi205.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQwfCfvL6+vsLobuCJ8jZNc8w06fscZOzxn6SeaIOq0Yh8KQH+8ukN1zBGdm5zb3ao5eKjSh8JgSiy9VnLgDmNja7gKLqJ+olpjVmlv5uDyumIIuAGWnT37dcT0AwMEmX9c+ZZEgapDHT5VhNxL/dPoOeimeGE8zJrh5pUCbW5GWZ74ZS4ipTUTC10h1VIy7hQ6Pl3FNEPtlK0KgAIzcgWbm+JLE+v0abIFJIsrCntZ+0Q+aE79lfvsSjaRxLpMUay/02/Onw/3JnzG8JWJTh+QkCmYbCIEObwdGuyA8idl4HpZHaPBgJNAt3qx22dcdgxNaO+TharP0SHYEQNIYU3++Kz+pTRpNG/V+L4GU1v1SUwIoiLL/amz+j9fCwQhi1D3S6ee00tJTD3d2opA+Lk9izf9Xk/89TkMSnnGknQ00huTWeJvBwHur7oB9r00Wx0c11HiigkNY+c0ESGTAXy9cA68XYLkDgTcPvsJkKyJXgtf+iiSpHN3mdDvk7IxoE= 2021-05-26T16:03:44.172 INFO:teuthology.kill:2021-05-26 16:03:44,170.170 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-26T16:03:44.172 INFO:teuthology.kill:2021-05-26 16:03:44,172.172 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-26T16:03:44.231 INFO:teuthology.kill:2021-05-26 16:03:44,231.231 INFO:teuthology.orchestra.console:Power off smithi205 2021-05-26T16:03:44.245 INFO:teuthology.kill:2021-05-26 16:03:44,245.245 INFO:teuthology.orchestra.console:Power off smithi137 2021-05-26T16:03:52.613 INFO:teuthology.kill:2021-05-26 16:03:52,613.613 INFO:teuthology.orchestra.console:Power off for smithi137 completed 2021-05-26T16:03:56.719 INFO:teuthology.kill:2021-05-26 16:03:56,719.719 INFO:teuthology.orchestra.console:Power off for smithi205 completed 2021-05-26T16:03:56.945 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2021-05-26T16:03:56.946 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2021-05-26T16:03:56.946 INFO:teuthology.misc:Compressing logs... 2021-05-26T16:03:56.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-26T16:04:31.563 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-26T16:04:31.631 INFO:teuthology.kill:No teuthology processes running 2021-05-26T16:04:32.720 INFO:teuthology.kill:Nuking machines: ['smithi137', 'smithi205'] 2021-05-26T16:04:33.653 INFO:teuthology.kill:2021-05-26 16:04:33,652.652 INFO:teuthology.nuke:targets: 2021-05-26T16:04:33.654 INFO:teuthology.kill: smithi137.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtlSkfeU5vTP+BuAtZdxvJxZ4RlaWR9oflAc5D5fSDJNrkvVEzuDBUsJVOTUJHZDBVCXUekqglJ5p8e3CnHhsQ4XjoY8Wa6dU16rCw8CohkSnfDf3HpDV3c2/gQDXv7Js6nZQnzm7Kw6OZNWaxjZIVX0GF67G7s7Xdlpz1+tUAK7GD6uVY0wKv3M6OXV3TSi4hgAaJbek3krbWAcfwxB6kTY9XL96J/u5jjs3e6iKv+5ltypjSZvEsnZndO7TFNMQbVTRlfzYXeh5DZm2Rn7CcAxH072ynX0wU/r92jSQYW/FltVoIfYXN+f6elG4q3MJZqApN7g/hbbpiCFBO1Wx0qbTOakNeLf/bbKIcMpBwJ5m8lffBWGDCz0XvyZWbzaaANjWNCOuz5NGORGYLMARRoZZyknh9d3ea63kqVVmPeFPlUl7cqkZv93k30PAgSHUMuxBLnLsxsZUEYr6Jz65tKbftXP5qBLvbFtEUzY1VCcVPL1Y+n5X3GggbGaT6C9s= 2021-05-26T16:04:33.654 INFO:teuthology.kill: smithi205.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQwfCfvL6+vsLobuCJ8jZNc8w06fscZOzxn6SeaIOq0Yh8KQH+8ukN1zBGdm5zb3ao5eKjSh8JgSiy9VnLgDmNja7gKLqJ+olpjVmlv5uDyumIIuAGWnT37dcT0AwMEmX9c+ZZEgapDHT5VhNxL/dPoOeimeGE8zJrh5pUCbW5GWZ74ZS4ipTUTC10h1VIy7hQ6Pl3FNEPtlK0KgAIzcgWbm+JLE+v0abIFJIsrCntZ+0Q+aE79lfvsSjaRxLpMUay/02/Onw/3JnzG8JWJTh+QkCmYbCIEObwdGuyA8idl4HpZHaPBgJNAt3qx22dcdgxNaO+TharP0SHYEQNIYU3++Kz+pTRpNG/V+L4GU1v1SUwIoiLL/amz+j9fCwQhi1D3S6ee00tJTD3d2opA+Lk9izf9Xk/89TkMSnnGknQ00huTWeJvBwHur7oB9r00Wx0c11HiigkNY+c0ESGTAXy9cA68XYLkDgTcPvsJkKyJXgtf+iiSpHN3mdDvk7IxoE= 2021-05-26T16:04:33.673 INFO:teuthology.kill:2021-05-26 16:04:33,673.673 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-26T16:04:33.676 INFO:teuthology.kill:2021-05-26 16:04:33,676.676 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-26T16:04:33.731 INFO:teuthology.kill:2021-05-26 16:04:33,731.731 INFO:teuthology.orchestra.console:Power off smithi137 2021-05-26T16:04:33.746 INFO:teuthology.kill:2021-05-26 16:04:33,746.746 INFO:teuthology.orchestra.console:Power off smithi205 2021-05-26T16:04:33.877 INFO:teuthology.kill:2021-05-26 16:04:33,876.876 INFO:teuthology.orchestra.console:Power off for smithi137 completed 2021-05-26T16:04:33.895 INFO:teuthology.kill:2021-05-26 16:04:33,895.895 INFO:teuthology.orchestra.console:Power off for smithi205 completed 2021-05-26T16:04:34.023 INFO:teuthology.kill:2021-05-26 16:04:34,022.022 INFO:teuthology.lock.ops:unlocked smithi137.front.sepia.ceph.com 2021-05-26T16:04:34.038 INFO:teuthology.kill:2021-05-26 16:04:34,038.038 INFO:teuthology.lock.ops:unlocked smithi205.front.sepia.ceph.com 2021-05-26T16:06:34.341 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-05-26T16:06:34.440 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-05-26T16:06:34.441 INFO:teuthology.nuke:Checking targets against current locks 2021-05-26T16:06:34.522 INFO:teuthology.nuke:Not nuking smithi137.front.sepia.ceph.com because description doesn't match 2021-05-26T16:06:34.522 INFO:teuthology.nuke:Not nuking smithi205.front.sepia.ceph.com because description doesn't match