2021-06-04T22:55:34.143 INFO:root:teuthology version: 1.1.0-f359b10d 2021-06-04T22:55:34.237 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi036.front.sepia.ceph.com' reimaging 2021-06-04T22:55:34.238 INFO:teuthology.lock.ops:Updating [ubuntu@smithi036.front.sepia.ceph.com]: reset os type and version on server 2021-06-04T22:55:34.238 INFO:teuthology.lock.ops:Updating smithi036.front.sepia.ceph.com on lock server 2021-06-04T22:55:34.258 INFO:teuthology.lock.ops:Node 'ubuntu@smithi036.front.sepia.ceph.com' reimaging is complete 2021-06-04T22:55:34.259 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi089.front.sepia.ceph.com' reimaging 2021-06-04T22:55:34.259 INFO:teuthology.lock.ops:Updating [ubuntu@smithi089.front.sepia.ceph.com]: reset os type and version on server 2021-06-04T22:55:34.260 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2021-06-04T22:55:34.282 INFO:teuthology.lock.ops:Node 'ubuntu@smithi089.front.sepia.ceph.com' reimaging is complete 2021-06-04T22:55:34.476 INFO:teuthology.provision.fog.smithi036:Scheduling deploy of ubuntu 20.04 2021-06-04T22:55:34.483 INFO:teuthology.provision.fog.smithi089:Scheduling deploy of ubuntu 20.04 2021-06-04T22:55:34.836 INFO:teuthology.orchestra.console:Power off smithi036 2021-06-04T22:55:34.850 INFO:teuthology.orchestra.console:Power off smithi089 2021-06-04T22:55:47.326 INFO:teuthology.orchestra.console:Power off for smithi036 completed 2021-06-04T22:55:47.332 INFO:teuthology.orchestra.console:Power off for smithi089 completed 2021-06-04T22:55:47.428 INFO:teuthology.orchestra.console:Power on smithi036 2021-06-04T22:55:47.437 INFO:teuthology.orchestra.console:Power on smithi089 2021-06-04T22:55:51.683 INFO:teuthology.orchestra.console:Power on for smithi036 completed 2021-06-04T22:55:51.689 INFO:teuthology.orchestra.console:Power on for smithi089 completed 2021-06-04T22:55:51.785 INFO:teuthology.provision.fog.smithi036:Waiting for deploy to finish 2021-06-04T22:55:51.790 INFO:teuthology.provision.fog.smithi089:Waiting for deploy to finish 2021-06-04T23:00:29.641 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-04T23:00:30.076 INFO:teuthology.provision.fog.smithi036:Node is ready 2021-06-04T23:00:30.083 INFO:teuthology.orchestra.run.smithi036.stdout:smithi036.front.sepia.ceph.com 2021-06-04T23:00:30.131 INFO:teuthology.orchestra.run.smithi036.stdout:172.21.15.36 smithi036.front.sepia.ceph.com smithi036 2021-06-04T23:00:30.312 INFO:teuthology.provision.fog.smithi036:Deploy complete! 2021-06-04T23:00:49.467 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-04T23:00:49.904 INFO:teuthology.provision.fog.smithi089:Node is ready 2021-06-04T23:00:49.910 INFO:teuthology.orchestra.run.smithi089.stdout:smithi089.front.sepia.ceph.com 2021-06-04T23:00:49.919 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89 smithi089.front.sepia.ceph.com smithi089 2021-06-04T23:00:50.172 INFO:teuthology.provision.fog.smithi089:Deploy complete! 2021-06-04T23:00:50.346 INFO:teuthology.lock.ops:Checking smithi036.front.sepia.ceph.com 2021-06-04T23:00:50.347 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-04T23:00:50.363 INFO:teuthology.lock.ops:Checking smithi089.front.sepia.ceph.com 2021-06-04T23:00:50.363 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-04T23:00:50.378 INFO:teuthology.lock.ops:Updating [smithi036.front.sepia.ceph.com]: set os type and version on server 2021-06-04T23:00:50.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-04T23:00:50.678 INFO:teuthology.orchestra.run.smithi036.stdout:x86_64 2021-06-04T23:00:50.686 INFO:teuthology.orchestra.run.smithi036.stdout:NAME="Ubuntu" 2021-06-04T23:00:50.687 INFO:teuthology.orchestra.run.smithi036.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-06-04T23:00:50.687 INFO:teuthology.orchestra.run.smithi036.stdout:ID=ubuntu 2021-06-04T23:00:50.687 INFO:teuthology.orchestra.run.smithi036.stdout:ID_LIKE=debian 2021-06-04T23:00:50.687 INFO:teuthology.orchestra.run.smithi036.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-06-04T23:00:50.688 INFO:teuthology.orchestra.run.smithi036.stdout:VERSION_ID="20.04" 2021-06-04T23:00:50.688 INFO:teuthology.orchestra.run.smithi036.stdout:HOME_URL="https://www.ubuntu.com/" 2021-06-04T23:00:50.688 INFO:teuthology.orchestra.run.smithi036.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-06-04T23:00:50.688 INFO:teuthology.orchestra.run.smithi036.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-06-04T23:00:50.689 INFO:teuthology.orchestra.run.smithi036.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-06-04T23:00:50.689 INFO:teuthology.orchestra.run.smithi036.stdout:VERSION_CODENAME=focal 2021-06-04T23:00:50.689 INFO:teuthology.orchestra.run.smithi036.stdout:UBUNTU_CODENAME=focal 2021-06-04T23:00:50.691 INFO:teuthology.lock.ops:Updating smithi036.front.sepia.ceph.com on lock server 2021-06-04T23:00:50.708 INFO:teuthology.lock.ops:Updating [smithi089.front.sepia.ceph.com]: set os type and version on server 2021-06-04T23:00:50.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-04T23:00:50.878 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2021-06-04T23:00:50.928 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="Ubuntu" 2021-06-04T23:00:50.929 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-06-04T23:00:50.929 INFO:teuthology.orchestra.run.smithi089.stdout:ID=ubuntu 2021-06-04T23:00:50.929 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE=debian 2021-06-04T23:00:50.930 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-06-04T23:00:50.930 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="20.04" 2021-06-04T23:00:50.930 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://www.ubuntu.com/" 2021-06-04T23:00:50.930 INFO:teuthology.orchestra.run.smithi089.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-06-04T23:00:50.931 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-06-04T23:00:50.931 INFO:teuthology.orchestra.run.smithi089.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-06-04T23:00:50.931 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_CODENAME=focal 2021-06-04T23:00:50.931 INFO:teuthology.orchestra.run.smithi089.stdout:UBUNTU_CODENAME=focal 2021-06-04T23:00:50.932 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2021-06-04T23:00:51.012 INFO:teuthology.dispatcher.supervisor:Running job 6152638 2021-06-04T23:00:51.013 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-06-04_18:17:44-fs-wip-yuri4-testing-2021-06-04-0748-pacific-distro-basic-smithi/6152638 --name yuriw-2021-06-04_18:17:44-fs-wip-yuri4-testing-2021-06-04-0748-pacific-distro-basic-smithi --description fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} -- /home/teuthworker/archive/yuriw-2021-06-04_18:17:44-fs-wip-yuri4-testing-2021-06-04-0748-pacific-distro-basic-smithi/6152638/orig.config.yaml 2021-06-04T23:00:51.017 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-06-04_18:17:44-fs-wip-yuri4-testing-2021-06-04-0748-pacific-distro-basic-smithi/6152638 2021-06-04T23:00:51.018 INFO:teuthology.dispatcher.supervisor:Job PID: 3636 2021-06-04T23:00:51.019 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-06-05T11:01:39.723 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-06-05T11:01:39.964 INFO:teuthology.kill:Killing Pids: {3636} 2021-06-05T11:01:41.105 INFO:teuthology.kill:Nuking machines: ['smithi036', 'smithi089'] 2021-06-05T11:01:42.109 INFO:teuthology.kill:2021-06-05 11:01:42,109.109 INFO:teuthology.nuke:targets: 2021-06-05T11:01:42.110 INFO:teuthology.kill: smithi036.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdYtyX7ou9vtypyrXtff9OGWTr7/e/ZyquVbpsq8EYiPIp9+F/f75f4BoN9RoJ1B+UxSUntMmpYNmhV04hrGQf44AGjRY+65YMr3gkj/7DjZfiIzDKFukMP/gI0G+R+4SGpJr5Psfqi291emtCS1BSCU7JUajuQ438r66oQ/DzTO2OvdEzKacQLipaPnCxUcnWb6d3Pc0xSFEE0AAGATLu5BmoneRVLu2MVam8zlSHqB4D1E9xaqAGaTrKbRa7P1XcJNcRu9efZegn70LLMgLWeum9EogCKHmdTLDjjcc4A8S1gDRRjVXPD55nIUIfItp1jcfC5JdRtqmORW02eIP9wT+a1k6mFgD2t+mQKrjQIhbVlXba8VOLy8OAVGejWg8EK3UUcaI7PY/GksF+nzDsHW9TdOTPwLEmznAgeYw3nHfFxAZzJU09Nt0too31dCEctw8dnD/ebPESXU4VIMEtqa07vrVj+aEFmNIHBe4LT3+3OANDNKFLA81aZRNQ3Y0= 2021-06-05T11:01:42.111 INFO:teuthology.kill: smithi089.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDD5ze2wZ6thIPoHB5y51aGSlj/6lKU9ETo1cPGLeN3dOJXik4gYAzvGI6RbL8m2GVicUyd0f4g7+B8gPpi8TuikNK6Nr0i/icfs0LY/roa8y3G83Qib20cscZYKyspUeUX+G4UnjLOtDRldA5QuF1gpTJwUnXcV4WJlnh6VJ6VdNLFYNasYuZRCHwA05IED4IC401CdrNL4HsTHFhnwGCqZ11x7V+gIeDNVcq7VxZr3V9G1cvFwtFuDEES3CDhin6jzTOzHQf+zwLPJ5K1rNfSqtqb6nCHKQjbJB8l+wF/2eZjs2uzj0UWmlc7Vn3WFfvVh+IToJsO3mZkJgx/H8JFbKc+wkCiRTQW+8uQDPiXkvJ9vvczJGaqw3mVUQaj7sbtiaY4zin85l5Z34KPx5Fu7blmAjSZkZrttlnXNhYLeAzephyzQFxPcWf/nIp6SelzUPL94fE80o1vGPwk75FzRzrfcGwfALp5SVFTObwrzaKBcuIIXEvDsa0CqnwqcTU= 2021-06-05T11:01:42.111 INFO:teuthology.kill:2021-06-05 11:01:42,110.110 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-05T11:01:42.112 INFO:teuthology.kill:2021-06-05 11:01:42,112.112 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-05T11:01:42.171 INFO:teuthology.kill:2021-06-05 11:01:42,171.171 INFO:teuthology.orchestra.console:Power off smithi036 2021-06-05T11:01:42.185 INFO:teuthology.kill:2021-06-05 11:01:42,185.185 INFO:teuthology.orchestra.console:Power off smithi089 2021-06-05T11:01:50.541 INFO:teuthology.kill:2021-06-05 11:01:50,541.541 INFO:teuthology.orchestra.console:Power off for smithi036 completed 2021-06-05T11:01:50.547 INFO:teuthology.kill:2021-06-05 11:01:50,546.546 INFO:teuthology.orchestra.console:Power off for smithi089 completed 2021-06-05T11:01:50.811 INFO:teuthology.task.internal:roles: ubuntu@smithi036.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2021-06-05T11:01:50.811 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2021-06-05T11:01:50.812 INFO:teuthology.misc:Compressing logs... 2021-06-05T11:01:50.812 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-05T11:02:25.483 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-06-05T11:02:25.555 INFO:teuthology.kill:No teuthology processes running 2021-06-05T11:02:26.658 INFO:teuthology.kill:Nuking machines: ['smithi036', 'smithi089'] 2021-06-05T11:02:27.607 INFO:teuthology.kill:2021-06-05 11:02:27,606.606 INFO:teuthology.nuke:targets: 2021-06-05T11:02:27.608 INFO:teuthology.kill: smithi036.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCdYtyX7ou9vtypyrXtff9OGWTr7/e/ZyquVbpsq8EYiPIp9+F/f75f4BoN9RoJ1B+UxSUntMmpYNmhV04hrGQf44AGjRY+65YMr3gkj/7DjZfiIzDKFukMP/gI0G+R+4SGpJr5Psfqi291emtCS1BSCU7JUajuQ438r66oQ/DzTO2OvdEzKacQLipaPnCxUcnWb6d3Pc0xSFEE0AAGATLu5BmoneRVLu2MVam8zlSHqB4D1E9xaqAGaTrKbRa7P1XcJNcRu9efZegn70LLMgLWeum9EogCKHmdTLDjjcc4A8S1gDRRjVXPD55nIUIfItp1jcfC5JdRtqmORW02eIP9wT+a1k6mFgD2t+mQKrjQIhbVlXba8VOLy8OAVGejWg8EK3UUcaI7PY/GksF+nzDsHW9TdOTPwLEmznAgeYw3nHfFxAZzJU09Nt0too31dCEctw8dnD/ebPESXU4VIMEtqa07vrVj+aEFmNIHBe4LT3+3OANDNKFLA81aZRNQ3Y0= 2021-06-05T11:02:27.608 INFO:teuthology.kill: smithi089.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDD5ze2wZ6thIPoHB5y51aGSlj/6lKU9ETo1cPGLeN3dOJXik4gYAzvGI6RbL8m2GVicUyd0f4g7+B8gPpi8TuikNK6Nr0i/icfs0LY/roa8y3G83Qib20cscZYKyspUeUX+G4UnjLOtDRldA5QuF1gpTJwUnXcV4WJlnh6VJ6VdNLFYNasYuZRCHwA05IED4IC401CdrNL4HsTHFhnwGCqZ11x7V+gIeDNVcq7VxZr3V9G1cvFwtFuDEES3CDhin6jzTOzHQf+zwLPJ5K1rNfSqtqb6nCHKQjbJB8l+wF/2eZjs2uzj0UWmlc7Vn3WFfvVh+IToJsO3mZkJgx/H8JFbKc+wkCiRTQW+8uQDPiXkvJ9vvczJGaqw3mVUQaj7sbtiaY4zin85l5Z34KPx5Fu7blmAjSZkZrttlnXNhYLeAzephyzQFxPcWf/nIp6SelzUPL94fE80o1vGPwk75FzRzrfcGwfALp5SVFTObwrzaKBcuIIXEvDsa0CqnwqcTU= 2021-06-05T11:02:27.626 INFO:teuthology.kill:2021-06-05 11:02:27,626.626 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-05T11:02:27.629 INFO:teuthology.kill:2021-06-05 11:02:27,629.629 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-05T11:02:27.686 INFO:teuthology.kill:2021-06-05 11:02:27,686.686 INFO:teuthology.orchestra.console:Power off smithi036 2021-06-05T11:02:27.699 INFO:teuthology.kill:2021-06-05 11:02:27,698.698 INFO:teuthology.orchestra.console:Power off smithi089 2021-06-05T11:02:27.828 INFO:teuthology.kill:2021-06-05 11:02:27,828.828 INFO:teuthology.orchestra.console:Power off for smithi036 completed 2021-06-05T11:02:27.840 INFO:teuthology.kill:2021-06-05 11:02:27,839.839 INFO:teuthology.orchestra.console:Power off for smithi089 completed 2021-06-05T11:02:27.970 INFO:teuthology.kill:2021-06-05 11:02:27,969.969 INFO:teuthology.lock.ops:unlocked smithi036.front.sepia.ceph.com 2021-06-05T11:02:27.980 INFO:teuthology.kill:2021-06-05 11:02:27,980.980 INFO:teuthology.lock.ops:unlocked smithi089.front.sepia.ceph.com 2021-06-05T11:04:28.282 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-06-05T11:04:28.378 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-06-05T11:04:28.379 INFO:teuthology.nuke:Checking targets against current locks 2021-06-05T11:04:28.450 INFO:teuthology.nuke:Not nuking smithi089.front.sepia.ceph.com because description doesn't match 2021-06-05T11:04:28.451 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology-dispatcher", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/scripts/dispatcher.py", line 33, in main teuthology.dispatcher.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/__init__.py", line 61, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 59, in main verbose File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 152, in run_job unlock_targets(job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 196, in unlock_targets nuke(fake_ctx, True) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable