2021-05-29T19:02:51.886 INFO:root:teuthology version: 1.1.0-f359b10d
2021-05-29T19:02:52.006 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi061.front.sepia.ceph.com' reimaging
2021-05-29T19:02:52.007 INFO:teuthology.lock.ops:Updating [ubuntu@smithi061.front.sepia.ceph.com]: reset os type and version on server
2021-05-29T19:02:52.008 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server
2021-05-29T19:02:52.024 INFO:teuthology.lock.ops:Node 'ubuntu@smithi061.front.sepia.ceph.com' reimaging is complete
2021-05-29T19:02:52.024 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi109.front.sepia.ceph.com' reimaging
2021-05-29T19:02:52.024 INFO:teuthology.lock.ops:Updating [ubuntu@smithi109.front.sepia.ceph.com]: reset os type and version on server
2021-05-29T19:02:52.025 INFO:teuthology.lock.ops:Updating smithi109.front.sepia.ceph.com on lock server
2021-05-29T19:02:52.044 INFO:teuthology.lock.ops:Node 'ubuntu@smithi109.front.sepia.ceph.com' reimaging is complete
2021-05-29T19:02:52.241 INFO:teuthology.provision.fog.smithi061:Scheduling deploy of rhel 8.3
2021-05-29T19:02:52.318 INFO:teuthology.provision.fog.smithi109:Scheduling deploy of rhel 8.3
2021-05-29T19:02:52.667 INFO:teuthology.orchestra.console:Power off smithi061
2021-05-29T19:02:52.757 INFO:teuthology.orchestra.console:Power off smithi109
2021-05-29T19:03:05.185 INFO:teuthology.orchestra.console:Power off for smithi061 completed
2021-05-29T19:03:05.250 INFO:teuthology.orchestra.console:Power off for smithi109 completed
2021-05-29T19:03:05.288 INFO:teuthology.orchestra.console:Power on smithi061
2021-05-29T19:03:05.351 INFO:teuthology.orchestra.console:Power on smithi109
2021-05-29T19:03:13.684 INFO:teuthology.orchestra.console:Power on for smithi061 completed
2021-05-29T19:03:13.715 INFO:teuthology.orchestra.console:Power on for smithi109 completed
2021-05-29T19:03:13.948 INFO:teuthology.provision.fog.smithi061:Waiting for deploy to finish
2021-05-29T19:03:14.048 INFO:teuthology.provision.fog.smithi109:Waiting for deploy to finish
2021-05-29T19:07:55.295 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-05-29T19:07:55.340 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-05-29T19:07:55.608 INFO:teuthology.provision.fog.smithi061:Node is ready
2021-05-29T19:07:55.623 INFO:teuthology.orchestra.run.smithi061.stdout:smithi061.front.sepia.ceph.com
2021-05-29T19:07:55.625 INFO:teuthology.provision.fog.smithi109:Node is ready
2021-05-29T19:07:55.639 INFO:teuthology.orchestra.run.smithi109.stdout:smithi109.front.sepia.ceph.com
2021-05-29T19:07:55.679 INFO:teuthology.orchestra.run.smithi061.stdout:172.21.15.61 smithi061.front.sepia.ceph.com smithi061
2021-05-29T19:07:55.695 INFO:teuthology.orchestra.run.smithi109.stdout:172.21.15.109 smithi109.front.sepia.ceph.com smithi109
2021-05-29T19:07:56.014 INFO:teuthology.provision.fog.smithi061:Deploy complete!
2021-05-29T19:07:56.024 INFO:teuthology.provision.fog.smithi109:Deploy complete!
2021-05-29T19:07:56.161 INFO:teuthology.lock.ops:Checking smithi061.front.sepia.ceph.com
2021-05-29T19:07:56.162 INFO:teuthology.lock.ops:New key found. Updating...
2021-05-29T19:07:56.182 INFO:teuthology.lock.ops:Checking smithi109.front.sepia.ceph.com
2021-05-29T19:07:56.183 INFO:teuthology.lock.ops:New key found. Updating...
2021-05-29T19:07:56.198 INFO:teuthology.lock.ops:Updating [smithi061.front.sepia.ceph.com]: set os type and version on server
2021-05-29T19:07:56.199 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-05-29T19:07:56.527 INFO:teuthology.orchestra.run.smithi061.stdout:x86_64
2021-05-29T19:07:56.582 INFO:teuthology.orchestra.run.smithi061.stdout:NAME="Red Hat Enterprise Linux"
2021-05-29T19:07:56.583 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION="8.3 (Ootpa)"
2021-05-29T19:07:56.583 INFO:teuthology.orchestra.run.smithi061.stdout:ID="rhel"
2021-05-29T19:07:56.584 INFO:teuthology.orchestra.run.smithi061.stdout:ID_LIKE="fedora"
2021-05-29T19:07:56.584 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_ID="8.3"
2021-05-29T19:07:56.584 INFO:teuthology.orchestra.run.smithi061.stdout:PLATFORM_ID="platform:el8"
2021-05-29T19:07:56.584 INFO:teuthology.orchestra.run.smithi061.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
2021-05-29T19:07:56.585 INFO:teuthology.orchestra.run.smithi061.stdout:ANSI_COLOR="0;31"
2021-05-29T19:07:56.585 INFO:teuthology.orchestra.run.smithi061.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
2021-05-29T19:07:56.585 INFO:teuthology.orchestra.run.smithi061.stdout:HOME_URL="https://www.redhat.com/"
2021-05-29T19:07:56.586 INFO:teuthology.orchestra.run.smithi061.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-05-29T19:07:56.586 INFO:teuthology.orchestra.run.smithi061.stdout:
2021-05-29T19:07:56.586 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2021-05-29T19:07:56.586 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
2021-05-29T19:07:56.587 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2021-05-29T19:07:56.587 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3"
2021-05-29T19:07:56.590 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server
2021-05-29T19:07:56.612 INFO:teuthology.lock.ops:Updating [smithi109.front.sepia.ceph.com]: set os type and version on server
2021-05-29T19:07:56.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-05-29T19:07:56.898 INFO:teuthology.orchestra.run.smithi109.stdout:x86_64
2021-05-29T19:07:56.957 INFO:teuthology.orchestra.run.smithi109.stdout:NAME="Red Hat Enterprise Linux"
2021-05-29T19:07:56.958 INFO:teuthology.orchestra.run.smithi109.stdout:VERSION="8.3 (Ootpa)"
2021-05-29T19:07:56.958 INFO:teuthology.orchestra.run.smithi109.stdout:ID="rhel"
2021-05-29T19:07:56.959 INFO:teuthology.orchestra.run.smithi109.stdout:ID_LIKE="fedora"
2021-05-29T19:07:56.959 INFO:teuthology.orchestra.run.smithi109.stdout:VERSION_ID="8.3"
2021-05-29T19:07:56.960 INFO:teuthology.orchestra.run.smithi109.stdout:PLATFORM_ID="platform:el8"
2021-05-29T19:07:56.960 INFO:teuthology.orchestra.run.smithi109.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
2021-05-29T19:07:56.961 INFO:teuthology.orchestra.run.smithi109.stdout:ANSI_COLOR="0;31"
2021-05-29T19:07:56.961 INFO:teuthology.orchestra.run.smithi109.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
2021-05-29T19:07:56.961 INFO:teuthology.orchestra.run.smithi109.stdout:HOME_URL="https://www.redhat.com/"
2021-05-29T19:07:56.963 INFO:teuthology.orchestra.run.smithi109.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-05-29T19:07:56.963 INFO:teuthology.orchestra.run.smithi109.stdout:
2021-05-29T19:07:56.963 INFO:teuthology.orchestra.run.smithi109.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2021-05-29T19:07:56.964 INFO:teuthology.orchestra.run.smithi109.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
2021-05-29T19:07:56.964 INFO:teuthology.orchestra.run.smithi109.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2021-05-29T19:07:56.965 INFO:teuthology.orchestra.run.smithi109.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3"
2021-05-29T19:07:56.966 INFO:teuthology.lock.ops:Updating smithi109.front.sepia.ceph.com on lock server
2021-05-29T19:07:57.127 INFO:teuthology.dispatcher.supervisor:Running job 6141891
2021-05-29T19:07:57.128 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology -v --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2021-05-29_16:04:00-rados-wip-sage3-testing-2021-05-29-1009-distro-basic-smithi/6141891 --name sage-2021-05-29_16:04:00-rados-wip-sage3-testing-2021-05-29-1009-distro-basic-smithi --description rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/snaps-few-objects} -- /home/teuthworker/archive/sage-2021-05-29_16:04:00-rados-wip-sage3-testing-2021-05-29-1009-distro-basic-smithi/6141891/orig.config.yaml
2021-05-29T19:07:57.135 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/sage-2021-05-29_16:04:00-rados-wip-sage3-testing-2021-05-29-1009-distro-basic-smithi/6141891
2021-05-29T19:07:57.137 INFO:teuthology.dispatcher.supervisor:Job PID: 8956
2021-05-29T19:07:57.138 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-05-30T07:08:45.552 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-05-30T07:08:45.741 INFO:teuthology.kill:Killing Pids: {8956}
2021-05-30T07:08:46.781 INFO:teuthology.kill:Nuking machines: ['smithi061', 'smithi109']
2021-05-30T07:08:47.602 INFO:teuthology.kill:2021-05-30 07:08:47,601.601 INFO:teuthology.nuke:targets:
2021-05-30T07:08:47.603 INFO:teuthology.kill:  smithi061.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqJbOjsNldFDqQpoE3flmkSfbt+g5fTlmcGlTFvD1uHC+Uvp73bbpcCgQxTiEuhkwB7gZBL/+z606xUB6HR4koxrHuk+kwiiIR75WkXAwmLNuqQNAYUvSLrFSYT036O5nsG/EcnTN8j5smwxxD8Sxx0zs5dMmkWDSExPJfWP4KzrjhjUUtM1UxFX2ztPWrqtv4KXVrX1elIReKvs8/tW3vcNB70F7G2eo87fgDgxgW47c2w2ijjLufj0k68u8ze8fbxyeSGw+IAJ7GrLxc5s0GCDVhcWfE5lFGlKR7SsRd0VIZmrFmjafV969Y/e+2LSBf1vwIs00jGRdaa8zg4DljBE/oTBxUdqd2WqBLJH7oFoQo/021t6Zw3/LBbdnISJUAttdu1UgAFfeJJ3FeRE1Fsw1HzTxosdgQdciZl960j4CGkycwnvu6xKpATkrGpFkfykFMugTyD7eVDeUjeGZuGLhFrSHaZLU2nfGMLZmjXtch1QqKsQe0rA9IUpWI+oE=
2021-05-30T07:08:47.603 INFO:teuthology.kill:  smithi109.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5DLbY/5/aJ0gGYMmqWUMdSP8L3S7eo8K+NDvwjUvRSIyKLis2b4muKUslXKs6b3JpnagfDiWbKsQcfZd0Yt2fq2CFLcyoFEJiazNC6Ti1gnAv/Ug+hCoQx7XXG8LH7FM7ppSbIZGcPnBLkkGsGyRkc4PS6+Ztz8d4UBT+hNEzc0GlRJWGSOEUjkj1AZe9DmGb80dtRi9rpxoHvXIJp0IaRnTGX99iZ1BIKRGh+hepHJezYkp9SIOMvorsL6+orB8lZQRgEMfZXiDsHo2Rei8BAdwqAniGSWvzwSbuvqzZRiV0knF8FiwagmOnPQHRrs98TXh14P4VQ0fNtuwMyUuFCieyGoM8lXo1s1IpOS1paBfZwPcCIEpP6v3+AN5mtd3HNFLb4QHSutzRGwezqYnp5l/XFY4YZLoknJfgZURsOi9jBXgMRESFQFjXL+tn5twMbESzOHnmq6lDYTiHg05neKcGPU0iO7t+rDfYnjgjUGDAJTEyc90A+P0CjQeHlw0=
2021-05-30T07:08:47.603 INFO:teuthology.kill:2021-05-30 07:08:47,602.602 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-30T07:08:47.604 INFO:teuthology.kill:2021-05-30 07:08:47,604.604 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-30T07:08:47.659 INFO:teuthology.kill:2021-05-30 07:08:47,659.659 INFO:teuthology.orchestra.console:Power off smithi061
2021-05-30T07:08:47.674 INFO:teuthology.kill:2021-05-30 07:08:47,673.673 INFO:teuthology.orchestra.console:Power off smithi109
2021-05-30T07:08:56.042 INFO:teuthology.kill:2021-05-30 07:08:56,042.042 INFO:teuthology.orchestra.console:Power off for smithi109 completed
2021-05-30T07:09:00.149 INFO:teuthology.kill:2021-05-30 07:09:00,149.149 INFO:teuthology.orchestra.console:Power off for smithi061 completed
2021-05-30T07:09:00.344 INFO:teuthology.task.internal:roles: ubuntu@smithi061.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2']
2021-05-30T07:09:00.345 INFO:teuthology.task.internal:roles: ubuntu@smithi109.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']
2021-05-30T07:09:00.345 INFO:teuthology.misc:Compressing logs...
2021-05-30T07:09:00.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-05-30T07:10:00.368 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in <listcomp>
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 434, in run
    transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-05-30T07:10:00.413 INFO:teuthology.kill:No teuthology processes running
2021-05-30T07:10:01.383 INFO:teuthology.kill:Nuking machines: ['smithi061', 'smithi109']
2021-05-30T07:10:02.231 INFO:teuthology.kill:2021-05-30 07:10:02,230.230 INFO:teuthology.nuke:targets:
2021-05-30T07:10:02.232 INFO:teuthology.kill:  smithi061.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqJbOjsNldFDqQpoE3flmkSfbt+g5fTlmcGlTFvD1uHC+Uvp73bbpcCgQxTiEuhkwB7gZBL/+z606xUB6HR4koxrHuk+kwiiIR75WkXAwmLNuqQNAYUvSLrFSYT036O5nsG/EcnTN8j5smwxxD8Sxx0zs5dMmkWDSExPJfWP4KzrjhjUUtM1UxFX2ztPWrqtv4KXVrX1elIReKvs8/tW3vcNB70F7G2eo87fgDgxgW47c2w2ijjLufj0k68u8ze8fbxyeSGw+IAJ7GrLxc5s0GCDVhcWfE5lFGlKR7SsRd0VIZmrFmjafV969Y/e+2LSBf1vwIs00jGRdaa8zg4DljBE/oTBxUdqd2WqBLJH7oFoQo/021t6Zw3/LBbdnISJUAttdu1UgAFfeJJ3FeRE1Fsw1HzTxosdgQdciZl960j4CGkycwnvu6xKpATkrGpFkfykFMugTyD7eVDeUjeGZuGLhFrSHaZLU2nfGMLZmjXtch1QqKsQe0rA9IUpWI+oE=
2021-05-30T07:10:02.232 INFO:teuthology.kill:  smithi109.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC5DLbY/5/aJ0gGYMmqWUMdSP8L3S7eo8K+NDvwjUvRSIyKLis2b4muKUslXKs6b3JpnagfDiWbKsQcfZd0Yt2fq2CFLcyoFEJiazNC6Ti1gnAv/Ug+hCoQx7XXG8LH7FM7ppSbIZGcPnBLkkGsGyRkc4PS6+Ztz8d4UBT+hNEzc0GlRJWGSOEUjkj1AZe9DmGb80dtRi9rpxoHvXIJp0IaRnTGX99iZ1BIKRGh+hepHJezYkp9SIOMvorsL6+orB8lZQRgEMfZXiDsHo2Rei8BAdwqAniGSWvzwSbuvqzZRiV0knF8FiwagmOnPQHRrs98TXh14P4VQ0fNtuwMyUuFCieyGoM8lXo1s1IpOS1paBfZwPcCIEpP6v3+AN5mtd3HNFLb4QHSutzRGwezqYnp5l/XFY4YZLoknJfgZURsOi9jBXgMRESFQFjXL+tn5twMbESzOHnmq6lDYTiHg05neKcGPU0iO7t+rDfYnjgjUGDAJTEyc90A+P0CjQeHlw0=
2021-05-30T07:10:02.252 INFO:teuthology.kill:2021-05-30 07:10:02,251.251 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-30T07:10:02.254 INFO:teuthology.kill:2021-05-30 07:10:02,254.254 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-30T07:10:02.308 INFO:teuthology.kill:2021-05-30 07:10:02,307.307 INFO:teuthology.orchestra.console:Power off smithi061
2021-05-30T07:10:02.320 INFO:teuthology.kill:2021-05-30 07:10:02,319.319 INFO:teuthology.orchestra.console:Power off smithi109
2021-05-30T07:10:02.447 INFO:teuthology.kill:2021-05-30 07:10:02,446.446 INFO:teuthology.orchestra.console:Power off for smithi061 completed
2021-05-30T07:10:02.458 INFO:teuthology.kill:2021-05-30 07:10:02,458.458 INFO:teuthology.orchestra.console:Power off for smithi109 completed
2021-05-30T07:10:02.585 INFO:teuthology.kill:2021-05-30 07:10:02,584.584 INFO:teuthology.lock.ops:unlocked smithi061.front.sepia.ceph.com
2021-05-30T07:10:02.594 INFO:teuthology.kill:2021-05-30 07:10:02,593.593 INFO:teuthology.lock.ops:unlocked smithi109.front.sepia.ceph.com
2021-05-30T07:12:02.781 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15