2021-02-17T12:55:50.336 INFO:root:teuthology version: 1.1.0-071a4df2
2021-02-17T12:55:50.546 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi172.front.sepia.ceph.com' reimaging
2021-02-17T12:55:50.548 INFO:teuthology.lock.ops:Updating [ubuntu@smithi172.front.sepia.ceph.com]: reset os type and version on server
2021-02-17T12:55:50.549 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server
2021-02-17T12:55:50.580 INFO:teuthology.lock.ops:Node 'ubuntu@smithi172.front.sepia.ceph.com' reimaging is complete
2021-02-17T12:55:50.581 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi187.front.sepia.ceph.com' reimaging
2021-02-17T12:55:50.584 INFO:teuthology.lock.ops:Updating [ubuntu@smithi187.front.sepia.ceph.com]: reset os type and version on server
2021-02-17T12:55:50.585 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server
2021-02-17T12:55:50.614 INFO:teuthology.lock.ops:Node 'ubuntu@smithi187.front.sepia.ceph.com' reimaging is complete
2021-02-17T12:55:50.822 INFO:teuthology.provision.fog.smithi172:Scheduling deploy of rhel 8.3
2021-02-17T12:55:50.894 INFO:teuthology.provision.fog.smithi187:Scheduling deploy of rhel 8.3
2021-02-17T12:55:51.186 INFO:teuthology.orchestra.console:Power off smithi172
2021-02-17T12:55:51.248 INFO:teuthology.orchestra.console:Power off smithi187
2021-02-17T12:56:03.806 INFO:teuthology.orchestra.console:Power off for smithi172 completed
2021-02-17T12:56:03.824 INFO:teuthology.orchestra.console:Power off for smithi187 completed
2021-02-17T12:56:03.911 INFO:teuthology.orchestra.console:Power on smithi172
2021-02-17T12:56:03.945 INFO:teuthology.orchestra.console:Power on smithi187
2021-02-17T12:56:12.371 INFO:teuthology.orchestra.console:Power on for smithi172 completed
2021-02-17T12:56:12.382 INFO:teuthology.orchestra.console:Power on for smithi187 completed
2021-02-17T12:56:12.480 INFO:teuthology.provision.fog.smithi172:Waiting for deploy to finish
2021-02-17T12:56:12.484 INFO:teuthology.provision.fog.smithi187:Waiting for deploy to finish
2021-02-17T13:00:53.527 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-02-17T13:00:53.843 INFO:teuthology.provision.fog.smithi187:Node is ready
2021-02-17T13:00:53.862 INFO:teuthology.orchestra.run.smithi187.stdout:smithi187.front.sepia.ceph.com
2021-02-17T13:00:53.917 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.15.187 smithi187.front.sepia.ceph.com smithi187
2021-02-17T13:00:54.255 INFO:teuthology.provision.fog.smithi187:Deploy complete!
2021-02-17T13:01:08.572 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-02-17T13:01:08.868 INFO:teuthology.provision.fog.smithi172:Node is ready
2021-02-17T13:01:08.887 INFO:teuthology.orchestra.run.smithi172.stdout:smithi172.front.sepia.ceph.com
2021-02-17T13:01:08.942 INFO:teuthology.orchestra.run.smithi172.stdout:172.21.15.172 smithi172.front.sepia.ceph.com smithi172
2021-02-17T13:01:09.256 INFO:teuthology.provision.fog.smithi172:Deploy complete!
2021-02-17T13:01:09.405 INFO:teuthology.lock.ops:Checking smithi172.front.sepia.ceph.com
2021-02-17T13:01:09.406 INFO:teuthology.lock.ops:New key found. Updating...
2021-02-17T13:01:09.480 INFO:teuthology.lock.ops:Checking smithi187.front.sepia.ceph.com
2021-02-17T13:01:09.480 INFO:teuthology.lock.ops:New key found. Updating...
2021-02-17T13:01:09.510 INFO:teuthology.lock.ops:Updating [smithi172.front.sepia.ceph.com]: set os type and version on server
2021-02-17T13:01:09.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-02-17T13:01:09.787 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64
2021-02-17T13:01:09.841 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="Red Hat Enterprise Linux"
2021-02-17T13:01:09.842 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="8.3 (Ootpa)"
2021-02-17T13:01:09.842 INFO:teuthology.orchestra.run.smithi172.stdout:ID="rhel"
2021-02-17T13:01:09.843 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="fedora"
2021-02-17T13:01:09.843 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="8.3"
2021-02-17T13:01:09.843 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el8"
2021-02-17T13:01:09.844 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
2021-02-17T13:01:09.844 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31"
2021-02-17T13:01:09.844 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
2021-02-17T13:01:09.844 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://www.redhat.com/"
2021-02-17T13:01:09.844 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-02-17T13:01:09.845 INFO:teuthology.orchestra.run.smithi172.stdout:
2021-02-17T13:01:09.845 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2021-02-17T13:01:09.845 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
2021-02-17T13:01:09.845 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2021-02-17T13:01:09.846 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3"
2021-02-17T13:01:09.847 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server
2021-02-17T13:01:09.864 INFO:teuthology.lock.ops:Updating [smithi187.front.sepia.ceph.com]: set os type and version on server
2021-02-17T13:01:09.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-02-17T13:01:10.156 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64
2021-02-17T13:01:10.213 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="Red Hat Enterprise Linux"
2021-02-17T13:01:10.214 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="8.3 (Ootpa)"
2021-02-17T13:01:10.214 INFO:teuthology.orchestra.run.smithi187.stdout:ID="rhel"
2021-02-17T13:01:10.215 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="fedora"
2021-02-17T13:01:10.215 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="8.3"
2021-02-17T13:01:10.216 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el8"
2021-02-17T13:01:10.216 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
2021-02-17T13:01:10.216 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31"
2021-02-17T13:01:10.216 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
2021-02-17T13:01:10.217 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://www.redhat.com/"
2021-02-17T13:01:10.217 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-02-17T13:01:10.217 INFO:teuthology.orchestra.run.smithi187.stdout:
2021-02-17T13:01:10.217 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2021-02-17T13:01:10.220 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
2021-02-17T13:01:10.220 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2021-02-17T13:01:10.221 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3"
2021-02-17T13:01:10.222 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server
2021-02-17T13:01:10.327 INFO:teuthology.dispatcher.supervisor:Running job 5890547
2021-02-17T13:01:10.327 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_071a4df2ce463088af781721173908ae84c1e2bd/virtualenv/bin/teuthology -v --owner scheduled_jcollin@teuthology --archive /home/teuthworker/archive/jcollin-2021-02-17_09:02:00-rados-wip-jcollin-dispatch-queue-throttle-test-Feb15-distro-basic-smithi/5890547 --name jcollin-2021-02-17_09:02:00-rados-wip-jcollin-dispatch-queue-throttle-test-Feb15-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-snaps} -- /home/teuthworker/archive/jcollin-2021-02-17_09:02:00-rados-wip-jcollin-dispatch-queue-throttle-test-Feb15-distro-basic-smithi/5890547/orig.config.yaml
2021-02-17T13:01:10.343 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/jcollin-2021-02-17_09:02:00-rados-wip-jcollin-dispatch-queue-throttle-test-Feb15-distro-basic-smithi/5890547
2021-02-17T13:01:10.346 INFO:teuthology.dispatcher.supervisor:Job PID: 8015
2021-02-17T13:01:10.346 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-02-18T01:01:26.231 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-02-18T01:01:41.040 INFO:teuthology.kill:Killing Pids: {8015}
2021-02-18T01:01:42.979 INFO:teuthology.kill:Nuking machines: ['smithi172', 'smithi187']
2021-02-18T01:01:44.873 INFO:teuthology.kill:2021-02-18 01:01:44,872.872 INFO:teuthology.nuke:targets:
2021-02-18T01:01:44.876 INFO:teuthology.kill:  smithi172.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfZ2KZyXQRUep+c4cojaF0ptsMdr+uRimBaELJ61bkJS8HC9PE6z2LRzqwlNhEbsyzqfrrytz5eH5BRoqTxDQofECdpx/2TZ64ZZmsdXoWf/tcYXgoNEHMAXHSmXGUsgLudesqFpWN/rPUnk4JJhroGT79JNdBNr4vPelJ8ryQddssXVGa0YdWV1haNm+yCn1Yek3XZDTqjXiC3sXc4XYmOW1c2/3a+0/9PZy8eir0/wY5c5GXffLzmjyfszfYo8muZOYKQZbX4beQp//oF+gi86TpnxbRt/pPAe3KIrI2nRowcY0Ezb+jYGWFCs8nmlW7eTBqTIRgh8y0mkSgUdktVai4QLFygPuenPXYQDvwISnnpYhCh/zi+bykgMgveZYDjNdaeZbS9sx5san95B5rXdrjV/46pfSRRn5rAH88k4jnb+qykHIoSneQ+N7y6gN9Y1dmOgDMGwgJ18tazwtk02rw3GKYHTqWFWNaVmRYb0kSc8WUR15+RHG5jSQWFPs=
2021-02-18T01:01:44.876 INFO:teuthology.kill:  smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjHQF23OPQwo0X+NyCNK101cHqBtS4bq8+IaJmKcccDdj227EwV0HIK3VJr6oiOz9BVHwfa+k5NUyjFfNdTKkQhMXAJpxx14sDk6lqkCha/OuLcuO+FULzlo68n3XNPNueOPzzG6PouxnNlAfGCGo0jDE0UE6mzCAXn1gcbuRI/vLKKr3Bi3XvVVgjNEmSScCRUwWhY8y8zvZGdKAacTuOqCO6SXtJpZHztfiWzfj6CR1vdDtt5iU1ENsOE89QUZBo1xGARJYunvw4VvQjYmAzEjcx0Kx5/YfFKIpODksh1sRzvqxW417GMzVKvRtXCy5vBYv7U5SZxd8WfKv8ZzW5mUna7UQgFoTSOsYSctTQB4JHX8DiW0xx7zLczEVE8prW3Gv3o0M8d8OKi2+njECiYlAoNXdxoEa0njcdATR6HrcksLMG1D69zvcDwcBmQkensjWERZAm2vUXfN01N6djhSUFWyEpEMR87ey0AWAdJou+/3/DHQINf3FkwWGM9Vc=
2021-02-18T01:01:44.877 INFO:teuthology.kill:2021-02-18 01:01:44,873.873 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-02-18T01:01:44.883 INFO:teuthology.kill:2021-02-18 01:01:44,883.883 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-02-18T01:01:44.954 INFO:teuthology.kill:2021-02-18 01:01:44,954.954 INFO:teuthology.orchestra.console:Power off smithi187
2021-02-18T01:01:44.972 INFO:teuthology.kill:2021-02-18 01:01:44,972.972 INFO:teuthology.orchestra.console:Power off smithi172
2021-02-18T01:01:57.459 INFO:teuthology.kill:2021-02-18 01:01:57,458.458 INFO:teuthology.orchestra.console:Power off for smithi187 completed
2021-02-18T01:01:57.469 INFO:teuthology.kill:2021-02-18 01:01:57,464.464 INFO:teuthology.orchestra.console:Power off for smithi172 completed
2021-02-18T01:01:57.721 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
2021-02-18T01:01:57.726 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
2021-02-18T01:01:57.726 INFO:teuthology.misc:Compressing logs...
2021-02-18T01:01:57.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-02-18T01:02:32.387 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_071a4df2ce463088af781721173908ae84c1e2bd/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_071a4df2ce463088af781721173908ae84c1e2bd/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_071a4df2ce463088af781721173908ae84c1e2bd/teuthology/misc.py", line 1372, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_071a4df2ce463088af781721173908ae84c1e2bd/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_071a4df2ce463088af781721173908ae84c1e2bd/teuthology/orchestra/cluster.py", line 64, in <listcomp>
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_071a4df2ce463088af781721173908ae84c1e2bd/teuthology/orchestra/remote.py", line 215, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_071a4df2ce463088af781721173908ae84c1e2bd/teuthology/orchestra/run.py", line 434, in run
    transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-02-18T01:02:32.458 INFO:teuthology.kill:No teuthology processes running
2021-02-18T01:02:34.464 INFO:teuthology.kill:Nuking machines: ['smithi172', 'smithi187']
2021-02-18T01:02:36.623 INFO:teuthology.kill:2021-02-18 01:02:36,623.623 INFO:teuthology.nuke:targets:
2021-02-18T01:02:36.628 INFO:teuthology.kill:  smithi172.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfZ2KZyXQRUep+c4cojaF0ptsMdr+uRimBaELJ61bkJS8HC9PE6z2LRzqwlNhEbsyzqfrrytz5eH5BRoqTxDQofECdpx/2TZ64ZZmsdXoWf/tcYXgoNEHMAXHSmXGUsgLudesqFpWN/rPUnk4JJhroGT79JNdBNr4vPelJ8ryQddssXVGa0YdWV1haNm+yCn1Yek3XZDTqjXiC3sXc4XYmOW1c2/3a+0/9PZy8eir0/wY5c5GXffLzmjyfszfYo8muZOYKQZbX4beQp//oF+gi86TpnxbRt/pPAe3KIrI2nRowcY0Ezb+jYGWFCs8nmlW7eTBqTIRgh8y0mkSgUdktVai4QLFygPuenPXYQDvwISnnpYhCh/zi+bykgMgveZYDjNdaeZbS9sx5san95B5rXdrjV/46pfSRRn5rAH88k4jnb+qykHIoSneQ+N7y6gN9Y1dmOgDMGwgJ18tazwtk02rw3GKYHTqWFWNaVmRYb0kSc8WUR15+RHG5jSQWFPs=
2021-02-18T01:02:36.628 INFO:teuthology.kill:  smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjHQF23OPQwo0X+NyCNK101cHqBtS4bq8+IaJmKcccDdj227EwV0HIK3VJr6oiOz9BVHwfa+k5NUyjFfNdTKkQhMXAJpxx14sDk6lqkCha/OuLcuO+FULzlo68n3XNPNueOPzzG6PouxnNlAfGCGo0jDE0UE6mzCAXn1gcbuRI/vLKKr3Bi3XvVVgjNEmSScCRUwWhY8y8zvZGdKAacTuOqCO6SXtJpZHztfiWzfj6CR1vdDtt5iU1ENsOE89QUZBo1xGARJYunvw4VvQjYmAzEjcx0Kx5/YfFKIpODksh1sRzvqxW417GMzVKvRtXCy5vBYv7U5SZxd8WfKv8ZzW5mUna7UQgFoTSOsYSctTQB4JHX8DiW0xx7zLczEVE8prW3Gv3o0M8d8OKi2+njECiYlAoNXdxoEa0njcdATR6HrcksLMG1D69zvcDwcBmQkensjWERZAm2vUXfN01N6djhSUFWyEpEMR87ey0AWAdJou+/3/DHQINf3FkwWGM9Vc=
2021-02-18T01:02:36.660 INFO:teuthology.kill:2021-02-18 01:02:36,660.660 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-02-18T01:02:36.676 INFO:teuthology.kill:2021-02-18 01:02:36,676.676 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-02-18T01:02:36.813 INFO:teuthology.kill:2021-02-18 01:02:36,812.812 INFO:teuthology.orchestra.console:Power off smithi187
2021-02-18T01:02:36.884 INFO:teuthology.kill:2021-02-18 01:02:36,884.884 INFO:teuthology.orchestra.console:Power off smithi172
2021-02-18T01:02:37.053 INFO:teuthology.kill:2021-02-18 01:02:37,053.053 INFO:teuthology.orchestra.console:Power off for smithi187 completed
2021-02-18T01:02:37.077 INFO:teuthology.kill:2021-02-18 01:02:37,077.077 INFO:teuthology.orchestra.console:Power off for smithi172 completed
2021-02-18T01:02:37.222 INFO:teuthology.kill:2021-02-18 01:02:37,221.221 INFO:teuthology.lock.ops:unlocked smithi187.front.sepia.ceph.com
2021-02-18T01:02:37.250 INFO:teuthology.kill:2021-02-18 01:02:37,250.250 INFO:teuthology.lock.ops:unlocked smithi172.front.sepia.ceph.com
2021-02-18T01:04:37.470 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-02-18T01:04:37.567 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2021-02-18T01:04:37.568 INFO:teuthology.nuke:Checking targets against current locks
2021-02-18T01:04:37.649 INFO:teuthology.nuke:Not nuking smithi187.front.sepia.ceph.com because description doesn't match
2021-02-18T01:04:37.649 INFO:teuthology.nuke:Not nuking smithi172.front.sepia.ceph.com because description doesn't match