2021-06-18T12:06:38.163 INFO:root:teuthology version: 1.1.0-f359b10d
2021-06-18T12:06:39.294 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi061.front.sepia.ceph.com' reimaging
2021-06-18T12:06:39.296 INFO:teuthology.lock.ops:Updating [ubuntu@smithi061.front.sepia.ceph.com]: reset os type and version on server
2021-06-18T12:06:39.296 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server
2021-06-18T12:06:39.311 INFO:teuthology.lock.ops:Node 'ubuntu@smithi061.front.sepia.ceph.com' reimaging is complete
2021-06-18T12:06:39.311 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi170.front.sepia.ceph.com' reimaging
2021-06-18T12:06:39.311 INFO:teuthology.lock.ops:Updating [ubuntu@smithi170.front.sepia.ceph.com]: reset os type and version on server
2021-06-18T12:06:39.312 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server
2021-06-18T12:06:39.328 INFO:teuthology.lock.ops:Node 'ubuntu@smithi170.front.sepia.ceph.com' reimaging is complete
2021-06-18T12:06:39.531 INFO:teuthology.provision.fog.smithi061:Scheduling deploy of centos 8.3
2021-06-18T12:06:39.563 INFO:teuthology.provision.fog.smithi170:Scheduling deploy of centos 8.3
2021-06-18T12:06:39.898 INFO:teuthology.orchestra.console:Power off smithi061
2021-06-18T12:06:39.974 INFO:teuthology.orchestra.console:Power off smithi170
2021-06-18T12:06:48.259 INFO:teuthology.orchestra.console:Power off for smithi061 completed
2021-06-18T12:06:48.334 INFO:teuthology.orchestra.console:Power off for smithi170 completed
2021-06-18T12:06:48.361 INFO:teuthology.orchestra.console:Power on smithi061
2021-06-18T12:06:48.436 INFO:teuthology.orchestra.console:Power on smithi170
2021-06-18T12:06:56.724 INFO:teuthology.orchestra.console:Power on for smithi061 completed
2021-06-18T12:06:56.792 INFO:teuthology.orchestra.console:Power on for smithi170 completed
2021-06-18T12:06:56.826 INFO:teuthology.provision.fog.smithi061:Waiting for deploy to finish
2021-06-18T12:06:56.893 INFO:teuthology.provision.fog.smithi170:Waiting for deploy to finish
2021-06-18T12:11:28.443 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-06-18T12:11:28.776 INFO:teuthology.provision.fog.smithi170:Node is ready
2021-06-18T12:11:28.790 INFO:teuthology.orchestra.run.smithi170.stdout:smithi170.front.sepia.ceph.com
2021-06-18T12:11:28.845 INFO:teuthology.orchestra.run.smithi170.stdout:172.21.15.170 smithi170.front.sepia.ceph.com smithi170
2021-06-18T12:11:29.174 INFO:teuthology.provision.fog.smithi170:Deploy complete!
2021-06-18T12:14:51.925 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-06-18T12:14:52.251 INFO:teuthology.provision.fog.smithi061:Node is ready
2021-06-18T12:14:52.266 INFO:teuthology.orchestra.run.smithi061.stdout:smithi061.front.sepia.ceph.com
2021-06-18T12:14:52.321 INFO:teuthology.orchestra.run.smithi061.stdout:172.21.15.61 smithi061.front.sepia.ceph.com smithi061
2021-06-18T12:14:52.653 INFO:teuthology.provision.fog.smithi061:Deploy complete!
2021-06-18T12:14:52.775 INFO:teuthology.lock.ops:Checking smithi061.front.sepia.ceph.com
2021-06-18T12:14:52.776 INFO:teuthology.lock.ops:New key found. Updating...
2021-06-18T12:14:52.793 INFO:teuthology.lock.ops:Checking smithi170.front.sepia.ceph.com
2021-06-18T12:14:52.793 INFO:teuthology.lock.ops:New key found. Updating...
2021-06-18T12:14:52.807 INFO:teuthology.lock.ops:Updating [smithi061.front.sepia.ceph.com]: set os type and version on server
2021-06-18T12:14:52.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-06-18T12:14:53.091 INFO:teuthology.orchestra.run.smithi061.stdout:x86_64
2021-06-18T12:14:53.145 INFO:teuthology.orchestra.run.smithi061.stdout:NAME="CentOS Linux"
2021-06-18T12:14:53.146 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION="8"
2021-06-18T12:14:53.146 INFO:teuthology.orchestra.run.smithi061.stdout:ID="centos"
2021-06-18T12:14:53.146 INFO:teuthology.orchestra.run.smithi061.stdout:ID_LIKE="rhel fedora"
2021-06-18T12:14:53.146 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_ID="8"
2021-06-18T12:14:53.147 INFO:teuthology.orchestra.run.smithi061.stdout:PLATFORM_ID="platform:el8"
2021-06-18T12:14:53.147 INFO:teuthology.orchestra.run.smithi061.stdout:PRETTY_NAME="CentOS Linux 8"
2021-06-18T12:14:53.147 INFO:teuthology.orchestra.run.smithi061.stdout:ANSI_COLOR="0;31"
2021-06-18T12:14:53.147 INFO:teuthology.orchestra.run.smithi061.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-06-18T12:14:53.148 INFO:teuthology.orchestra.run.smithi061.stdout:HOME_URL="https://centos.org/"
2021-06-18T12:14:53.148 INFO:teuthology.orchestra.run.smithi061.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-06-18T12:14:53.148 INFO:teuthology.orchestra.run.smithi061.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-06-18T12:14:53.148 INFO:teuthology.orchestra.run.smithi061.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-06-18T12:14:53.150 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server
2021-06-18T12:14:53.165 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: set os type and version on server
2021-06-18T12:14:53.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-06-18T12:14:53.383 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64
2021-06-18T12:14:53.438 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Linux"
2021-06-18T12:14:53.439 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="8"
2021-06-18T12:14:53.439 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos"
2021-06-18T12:14:53.440 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora"
2021-06-18T12:14:53.440 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="8"
2021-06-18T12:14:53.440 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el8"
2021-06-18T12:14:53.441 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Linux 8"
2021-06-18T12:14:53.441 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31"
2021-06-18T12:14:53.442 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-06-18T12:14:53.442 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/"
2021-06-18T12:14:53.443 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugs.centos.org/"
2021-06-18T12:14:53.443 INFO:teuthology.orchestra.run.smithi170.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8"
2021-06-18T12:14:53.443 INFO:teuthology.orchestra.run.smithi170.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8"
2021-06-18T12:14:53.444 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server
2021-06-18T12:14:53.553 INFO:teuthology.dispatcher.supervisor:Running job 6178546
2021-06-18T12:14:53.553 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology -v --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2021-06-18_12:04:35-rados-wip-kefu-testing-2021-06-18-1414-distro-basic-smithi/6178546 --name kchai-2021-06-18_12:04:35-rados-wip-kefu-testing-2021-06-18-1414-distro-basic-smithi --description rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/force-sync-many workloads/snaps-few-objects} -- /home/teuthworker/archive/kchai-2021-06-18_12:04:35-rados-wip-kefu-testing-2021-06-18-1414-distro-basic-smithi/6178546/orig.config.yaml
2021-06-18T12:14:53.557 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/kchai-2021-06-18_12:04:35-rados-wip-kefu-testing-2021-06-18-1414-distro-basic-smithi/6178546
2021-06-18T12:14:53.557 INFO:teuthology.dispatcher.supervisor:Job PID: 22329
2021-06-18T12:14:53.558 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-06-19T00:15:46.663 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-06-19T00:15:46.880 INFO:teuthology.kill:Killing Pids: {22329}
2021-06-19T00:15:48.024 INFO:teuthology.kill:Nuking machines: ['smithi061', 'smithi170']
2021-06-19T00:15:48.975 INFO:teuthology.kill:2021-06-19 00:15:48,975.975 INFO:teuthology.nuke:targets:
2021-06-19T00:15:48.976 INFO:teuthology.kill:  smithi061.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmaFYUgcLQGOJhlp4TdCko7NHR4IN3aqq+Wr3xZoY6L1KqeoBEgOtmQBRBVLwmcKWljXtXcDBUAaP+oCTgsq1jpKJqzxjeDdDpyPPHwE+2hSpGZ2GrNJz6peAVKiJwwcz3Qngth4hYMxdwb5+Q0+Ak5GP5lyaemkeZ6zwZaqK72+t2zfRslj8FrN7c8mD+9QZx41CANB8Hn5ICmGLzNf9asKWhsjBInsmFHKCS9S28xVMgO96o1U1tFU5vIJIdX3tFhASLfybzljxuXBR8s82fkO/YWGqle75Usvy0eVqIHFw+ZPlQWfyZA2X3993BZI0W6iXbQzev3ou4L8eOfK18AQwdsrmbIwlPUThOor5UqntV78FAP3kURP8CHlyfBJOOJkbaumeoHF8HhnOld2Qu7DRjcjMuN09hX0kYP2QMiJHKc8QIqjveMMDusLcAEe1NJZg9C4bLEhMCqtLsH0DcaAcwxv72449Y6XrZNQTxLdhAWoKobXFYKxzs2CZbLI0=
2021-06-19T00:15:48.976 INFO:teuthology.kill:  smithi170.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4aUdNQkGsJRszH7pWe5mlOK+ckhMRPOsmCONFfOAIte8dUho+f2f45PdW1wgLN56lfWdaZyQ5hACjiP7tzE1ILpAvCnLA9YEGBJVcWbqtinYqDZIm4/2SoUkN8u0KHixe+qvLCL19AOhs0B96NcvLBrrPfDkuHpvZVPGXDipYsRmcJ5XyCmZ0QcGa0mlbFF0i+3QJY84kcwdaUm4FMxz3kZvVrUIx86CBXkAU2ZnlosX+AC3mTcbbGlPMoq8ApP7A/W/VcRwaVCL1/5tpVK2HswMqiisnl1ksC5XkRwX2YMeP2X1uqokJkMh2t/chaSW9JPmxxR61/0xPU7nWqh8wZHdD5YryNxYo9CzqKyVNeF6MezbqdCR4KwpSPkX6K4f/FmdUGIQ4MtlAUDwuYMdBHKGYkibWfV1mooJvSZTJr1xgYa3Hz1tvoCtiS5oCYMN0k3DRYQt+I7SXBjhQ/Bvj6IpRIVfCjuO6iHU3de7sgdsHvjBT9XjtDgSg6461jts=
2021-06-19T00:15:48.977 INFO:teuthology.kill:2021-06-19 00:15:48,975.975 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-06-19T00:15:48.977 INFO:teuthology.kill:2021-06-19 00:15:48,977.977 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-06-19T00:15:49.033 INFO:teuthology.kill:2021-06-19 00:15:49,033.033 INFO:teuthology.orchestra.console:Power off smithi061
2021-06-19T00:15:49.045 INFO:teuthology.kill:2021-06-19 00:15:49,044.044 INFO:teuthology.orchestra.console:Power off smithi170
2021-06-19T00:15:57.405 INFO:teuthology.kill:2021-06-19 00:15:57,404.404 INFO:teuthology.orchestra.console:Power off for smithi061 completed
2021-06-19T00:15:57.410 INFO:teuthology.kill:2021-06-19 00:15:57,410.410 INFO:teuthology.orchestra.console:Power off for smithi170 completed
2021-06-19T00:15:57.661 INFO:teuthology.task.internal:roles: ubuntu@smithi061.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2']
2021-06-19T00:15:57.661 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']
2021-06-19T00:15:57.662 INFO:teuthology.misc:Compressing logs...
2021-06-19T00:15:57.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-06-19T00:16:57.724 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in <listcomp>
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 434, in run
    transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-06-19T00:16:57.776 INFO:teuthology.kill:No teuthology processes running
2021-06-19T00:16:58.795 INFO:teuthology.kill:Nuking machines: ['smithi061', 'smithi170']
2021-06-19T00:16:59.701 INFO:teuthology.kill:2021-06-19 00:16:59,701.701 INFO:teuthology.nuke:targets:
2021-06-19T00:16:59.702 INFO:teuthology.kill:  smithi061.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmaFYUgcLQGOJhlp4TdCko7NHR4IN3aqq+Wr3xZoY6L1KqeoBEgOtmQBRBVLwmcKWljXtXcDBUAaP+oCTgsq1jpKJqzxjeDdDpyPPHwE+2hSpGZ2GrNJz6peAVKiJwwcz3Qngth4hYMxdwb5+Q0+Ak5GP5lyaemkeZ6zwZaqK72+t2zfRslj8FrN7c8mD+9QZx41CANB8Hn5ICmGLzNf9asKWhsjBInsmFHKCS9S28xVMgO96o1U1tFU5vIJIdX3tFhASLfybzljxuXBR8s82fkO/YWGqle75Usvy0eVqIHFw+ZPlQWfyZA2X3993BZI0W6iXbQzev3ou4L8eOfK18AQwdsrmbIwlPUThOor5UqntV78FAP3kURP8CHlyfBJOOJkbaumeoHF8HhnOld2Qu7DRjcjMuN09hX0kYP2QMiJHKc8QIqjveMMDusLcAEe1NJZg9C4bLEhMCqtLsH0DcaAcwxv72449Y6XrZNQTxLdhAWoKobXFYKxzs2CZbLI0=
2021-06-19T00:16:59.702 INFO:teuthology.kill:  smithi170.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4aUdNQkGsJRszH7pWe5mlOK+ckhMRPOsmCONFfOAIte8dUho+f2f45PdW1wgLN56lfWdaZyQ5hACjiP7tzE1ILpAvCnLA9YEGBJVcWbqtinYqDZIm4/2SoUkN8u0KHixe+qvLCL19AOhs0B96NcvLBrrPfDkuHpvZVPGXDipYsRmcJ5XyCmZ0QcGa0mlbFF0i+3QJY84kcwdaUm4FMxz3kZvVrUIx86CBXkAU2ZnlosX+AC3mTcbbGlPMoq8ApP7A/W/VcRwaVCL1/5tpVK2HswMqiisnl1ksC5XkRwX2YMeP2X1uqokJkMh2t/chaSW9JPmxxR61/0xPU7nWqh8wZHdD5YryNxYo9CzqKyVNeF6MezbqdCR4KwpSPkX6K4f/FmdUGIQ4MtlAUDwuYMdBHKGYkibWfV1mooJvSZTJr1xgYa3Hz1tvoCtiS5oCYMN0k3DRYQt+I7SXBjhQ/Bvj6IpRIVfCjuO6iHU3de7sgdsHvjBT9XjtDgSg6461jts=
2021-06-19T00:16:59.720 INFO:teuthology.kill:2021-06-19 00:16:59,719.719 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-06-19T00:16:59.722 INFO:teuthology.kill:2021-06-19 00:16:59,722.722 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-06-19T00:16:59.773 INFO:teuthology.kill:2021-06-19 00:16:59,773.773 INFO:teuthology.orchestra.console:Power off smithi061
2021-06-19T00:16:59.786 INFO:teuthology.kill:2021-06-19 00:16:59,786.786 INFO:teuthology.orchestra.console:Power off smithi170
2021-06-19T00:16:59.915 INFO:teuthology.kill:2021-06-19 00:16:59,915.915 INFO:teuthology.orchestra.console:Power off for smithi061 completed
2021-06-19T00:16:59.928 INFO:teuthology.kill:2021-06-19 00:16:59,927.927 INFO:teuthology.orchestra.console:Power off for smithi170 completed
2021-06-19T00:17:00.057 INFO:teuthology.kill:2021-06-19 00:17:00,055.055 INFO:teuthology.lock.ops:unlocked smithi061.front.sepia.ceph.com
2021-06-19T00:17:00.064 INFO:teuthology.kill:2021-06-19 00:17:00,063.063 INFO:teuthology.lock.ops:unlocked smithi170.front.sepia.ceph.com
2021-06-19T00:19:00.325 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-06-19T00:19:00.404 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2021-06-19T00:19:00.404 INFO:teuthology.nuke:Checking targets against current locks
2021-06-19T00:19:00.473 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology-dispatcher", line 33, in <module>
    sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-dispatcher')())
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/scripts/dispatcher.py", line 33, in main
    teuthology.dispatcher.main(args)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/__init__.py", line 61, in main
    return supervisor.main(args)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 59, in main
    verbose
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 152, in run_job
    unlock_targets(job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 196, in unlock_targets
    nuke(fake_ctx, True)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/nuke/__init__.py", line 250, in nuke
    if ctx.name not in lock['description']:
TypeError: argument of type 'NoneType' is not iterable