2021-04-23T09:55:06.377 INFO:root:teuthology version: 1.1.0-2713a3cd
2021-04-23T09:55:06.570 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi142.front.sepia.ceph.com' reimaging
2021-04-23T09:55:06.570 INFO:teuthology.lock.ops:Updating [ubuntu@smithi142.front.sepia.ceph.com]: reset os type and version on server
2021-04-23T09:55:06.570 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server
2021-04-23T09:55:06.669 INFO:teuthology.lock.ops:Node 'ubuntu@smithi142.front.sepia.ceph.com' reimaging is complete
2021-04-23T09:55:06.669 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi198.front.sepia.ceph.com' reimaging
2021-04-23T09:55:06.669 INFO:teuthology.lock.ops:Updating [ubuntu@smithi198.front.sepia.ceph.com]: reset os type and version on server
2021-04-23T09:55:06.669 INFO:teuthology.lock.ops:Updating smithi198.front.sepia.ceph.com on lock server
2021-04-23T09:55:06.715 INFO:teuthology.lock.ops:Node 'ubuntu@smithi198.front.sepia.ceph.com' reimaging is complete
2021-04-23T09:55:06.964 INFO:teuthology.provision.fog.smithi142:Scheduling deploy of rhel 8.3
2021-04-23T09:55:06.978 INFO:teuthology.provision.fog.smithi198:Scheduling deploy of rhel 8.3
2021-04-23T09:55:07.444 INFO:teuthology.orchestra.console:Power off smithi142
2021-04-23T09:55:07.515 INFO:teuthology.orchestra.console:Power off smithi198
2021-04-23T09:55:15.823 INFO:teuthology.orchestra.console:Power off for smithi142 completed
2021-04-23T09:55:15.924 INFO:teuthology.orchestra.console:Power on smithi142
2021-04-23T09:55:19.997 INFO:teuthology.orchestra.console:Power off for smithi198 completed
2021-04-23T09:55:20.099 INFO:teuthology.orchestra.console:Power on smithi198
2021-04-23T09:55:24.286 INFO:teuthology.orchestra.console:Power on for smithi142 completed
2021-04-23T09:55:24.350 INFO:teuthology.orchestra.console:Power on for smithi198 completed
2021-04-23T09:55:24.387 INFO:teuthology.provision.fog.smithi142:Waiting for deploy to finish
2021-04-23T09:55:24.451 INFO:teuthology.provision.fog.smithi198:Waiting for deploy to finish
2021-04-23T09:59:30.669 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-04-23T09:59:50.337 INFO:teuthology.provision.fog.smithi198:Node is ready
2021-04-23T09:59:50.356 INFO:teuthology.orchestra.run.smithi198.stdout:smithi198.front.sepia.ceph.com
2021-04-23T09:59:50.412 INFO:teuthology.orchestra.run.smithi198.stdout:172.21.15.198 smithi198.front.sepia.ceph.com smithi198
2021-04-23T09:59:50.766 INFO:teuthology.provision.fog.smithi198:Deploy complete!
2021-04-23T09:59:56.254 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-04-23T09:59:56.566 INFO:teuthology.provision.fog.smithi142:Node is ready
2021-04-23T09:59:56.581 INFO:teuthology.orchestra.run.smithi142.stdout:smithi142.front.sepia.ceph.com
2021-04-23T09:59:56.636 INFO:teuthology.orchestra.run.smithi142.stdout:172.21.15.142 smithi142.front.sepia.ceph.com smithi142
2021-04-23T09:59:56.958 INFO:teuthology.provision.fog.smithi142:Deploy complete!
2021-04-23T09:59:57.079 INFO:teuthology.lock.ops:Checking smithi142.front.sepia.ceph.com
2021-04-23T09:59:57.080 INFO:teuthology.lock.ops:New key found. Updating...
2021-04-23T09:59:57.097 INFO:teuthology.lock.ops:Checking smithi198.front.sepia.ceph.com
2021-04-23T09:59:57.097 INFO:teuthology.lock.ops:New key found. Updating...
2021-04-23T09:59:57.113 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: set os type and version on server
2021-04-23T09:59:57.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-04-23T09:59:57.387 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64
2021-04-23T09:59:57.441 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="Red Hat Enterprise Linux"
2021-04-23T09:59:57.441 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="8.3 (Ootpa)"
2021-04-23T09:59:57.441 INFO:teuthology.orchestra.run.smithi142.stdout:ID="rhel"
2021-04-23T09:59:57.441 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="fedora"
2021-04-23T09:59:57.441 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="8.3"
2021-04-23T09:59:57.442 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el8"
2021-04-23T09:59:57.442 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
2021-04-23T09:59:57.442 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31"
2021-04-23T09:59:57.442 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
2021-04-23T09:59:57.442 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://www.redhat.com/"
2021-04-23T09:59:57.442 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-04-23T09:59:57.442 INFO:teuthology.orchestra.run.smithi142.stdout:
2021-04-23T09:59:57.442 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2021-04-23T09:59:57.442 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
2021-04-23T09:59:57.442 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2021-04-23T09:59:57.442 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3"
2021-04-23T09:59:57.443 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server
2021-04-23T09:59:57.461 INFO:teuthology.lock.ops:Updating [smithi198.front.sepia.ceph.com]: set os type and version on server
2021-04-23T09:59:57.461 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-04-23T09:59:57.791 INFO:teuthology.orchestra.run.smithi198.stdout:x86_64
2021-04-23T09:59:57.845 INFO:teuthology.orchestra.run.smithi198.stdout:NAME="Red Hat Enterprise Linux"
2021-04-23T09:59:57.846 INFO:teuthology.orchestra.run.smithi198.stdout:VERSION="8.3 (Ootpa)"
2021-04-23T09:59:57.846 INFO:teuthology.orchestra.run.smithi198.stdout:ID="rhel"
2021-04-23T09:59:57.846 INFO:teuthology.orchestra.run.smithi198.stdout:ID_LIKE="fedora"
2021-04-23T09:59:57.846 INFO:teuthology.orchestra.run.smithi198.stdout:VERSION_ID="8.3"
2021-04-23T09:59:57.846 INFO:teuthology.orchestra.run.smithi198.stdout:PLATFORM_ID="platform:el8"
2021-04-23T09:59:57.846 INFO:teuthology.orchestra.run.smithi198.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
2021-04-23T09:59:57.846 INFO:teuthology.orchestra.run.smithi198.stdout:ANSI_COLOR="0;31"
2021-04-23T09:59:57.846 INFO:teuthology.orchestra.run.smithi198.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
2021-04-23T09:59:57.846 INFO:teuthology.orchestra.run.smithi198.stdout:HOME_URL="https://www.redhat.com/"
2021-04-23T09:59:57.847 INFO:teuthology.orchestra.run.smithi198.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-04-23T09:59:57.847 INFO:teuthology.orchestra.run.smithi198.stdout:
2021-04-23T09:59:57.847 INFO:teuthology.orchestra.run.smithi198.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2021-04-23T09:59:57.847 INFO:teuthology.orchestra.run.smithi198.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
2021-04-23T09:59:57.847 INFO:teuthology.orchestra.run.smithi198.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2021-04-23T09:59:57.847 INFO:teuthology.orchestra.run.smithi198.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3"
2021-04-23T09:59:57.848 INFO:teuthology.lock.ops:Updating smithi198.front.sepia.ceph.com on lock server
2021-04-23T09:59:57.923 INFO:teuthology.dispatcher.supervisor:Running job 6064377
2021-04-23T09:59:57.924 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2021-04-22_11:34:56-rados-no_column_b-distro-basic-smithi/6064377 --name benhanokh-2021-04-22_11:34:56-rados-no_column_b-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-localized} -- /home/teuthworker/archive/benhanokh-2021-04-22_11:34:56-rados-no_column_b-distro-basic-smithi/6064377/orig.config.yaml
2021-04-23T09:59:57.929 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2021-04-22_11:34:56-rados-no_column_b-distro-basic-smithi/6064377
2021-04-23T09:59:57.930 INFO:teuthology.dispatcher.supervisor:Job PID: 13246
2021-04-23T09:59:57.930 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-04-23T22:01:03.496 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-04-23T22:01:03.661 INFO:teuthology.kill:Killing Pids: {13246}
2021-04-23T22:01:04.834 INFO:teuthology.kill:Nuking machines: ['smithi142', 'smithi198']
2021-04-23T22:01:05.785 INFO:teuthology.kill:2021-04-23 22:01:05,785.785 INFO:teuthology.nuke:targets:
2021-04-23T22:01:05.786 INFO:teuthology.kill:  smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/CvMsvglYAkgnK0T+8pVrGQfHOizs6ik++LmAFY9YWxcxxczNotkHsQejmFWUmA3403CoLxuXXnt6LXikwf/TJHG9caSU2v7STsSGhHHRAa5JICTBnsycqZAS8ZDIKEaFMnGV7GSnNn0nAcqFkdJL+khOkBQb4IGBpMIB9qLx7XsD/4DqcVALzksdR2SZaR3QLqU2dlEEO+8EpPX7K1tkmv7V49y8CFWS7OzDYY/hBzBlpKJ8KYY6/ykiJlBTEPfYwvKRF3Tdh3SJIjhisAEVia3ka7jphe2tgsu0Rl6Djvl+gPxGCIdHk9YRowwWS6N3TsHid9tQN11/lSJ2PzSNQSRlAG0xKHxBleIQzZRekjvMdAlDfDNCEPM/viwgIdShpJxjlG+iv8uoaTtiRFGfi3AZx4N6a8dSji/9WyJUKtOhGGJLx2kKTZ2Syu8uhwrjJJz5HN47u7D/Ox3In6qgJpcLnYNX5x18tyxBimQ6vQa6LtCzpM92tO4kOnxnI8c=
2021-04-23T22:01:05.786 INFO:teuthology.kill:  smithi198.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaeNR48KC4TcQ2aCRNowcUVbFtjyuDBBHYur6XR/BHZccqju1W7805xid2kipJvY+ZW8W6jxVjeOFkDyEwrrVqazljjr1PtrkprI93PqnTPEoSO77LCNP/vxtobhjAM9sxxoDG8UaOdSL73QCNzmirV+t4MZcc5h+H4AHbhlOl5v+21i3DabG3B9F2g4eBoPgnQyIEVdM4IWAUa6zeK8nO5w044TISv1sB7ciVqmhmQieuD97Uliwqbzkgj4w5GTc2kJ606a57GjIUhZfXCDqjtbicZPDYGKxTOthjFZiWy4BOPJKPjVlawOKDE9Zhsr9WoGl0FsPVKNDlMP6hQ4KqmsS9b7KN70SJkX5nYBSuCMtafhmrVD46HaGXpQNfiISxdz3w/xSGiW3Te9HdVorpqLf0ODM9I15VyyTRpcdu1CL1IhM6+cJAM02VNNGij82XAngQ4u5oqm/Hy6fkKnDhUxjzPms8mea1xJj7Mc9/1ZRjPufIsOXXfIQkoh5zn2s=
2021-04-23T22:01:05.786 INFO:teuthology.kill:2021-04-23 22:01:05,785.785 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-23T22:01:05.788 INFO:teuthology.kill:2021-04-23 22:01:05,788.788 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-23T22:01:05.861 INFO:teuthology.kill:2021-04-23 22:01:05,861.861 INFO:teuthology.orchestra.console:Power off smithi142
2021-04-23T22:01:05.874 INFO:teuthology.kill:2021-04-23 22:01:05,874.874 INFO:teuthology.orchestra.console:Power off smithi198
2021-04-23T22:01:14.244 INFO:teuthology.kill:2021-04-23 22:01:14,243.243 INFO:teuthology.orchestra.console:Power off for smithi198 completed
2021-04-23T22:01:18.351 INFO:teuthology.kill:2021-04-23 22:01:18,351.351 INFO:teuthology.orchestra.console:Power off for smithi142 completed
2021-04-23T22:01:18.574 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
2021-04-23T22:01:18.574 INFO:teuthology.task.internal:roles: ubuntu@smithi198.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
2021-04-23T22:01:18.575 INFO:teuthology.misc:Compressing logs...
2021-04-23T22:01:18.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-04-23T22:01:53.247 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in <listcomp>
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/remote.py", line 215, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/run.py", line 434, in run
    transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-04-23T22:01:53.311 INFO:teuthology.kill:No teuthology processes running
2021-04-23T22:01:54.655 INFO:teuthology.kill:Nuking machines: ['smithi142', 'smithi198']
2021-04-23T22:01:55.619 INFO:teuthology.kill:2021-04-23 22:01:55,618.618 INFO:teuthology.nuke:targets:
2021-04-23T22:01:55.619 INFO:teuthology.kill:  smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/CvMsvglYAkgnK0T+8pVrGQfHOizs6ik++LmAFY9YWxcxxczNotkHsQejmFWUmA3403CoLxuXXnt6LXikwf/TJHG9caSU2v7STsSGhHHRAa5JICTBnsycqZAS8ZDIKEaFMnGV7GSnNn0nAcqFkdJL+khOkBQb4IGBpMIB9qLx7XsD/4DqcVALzksdR2SZaR3QLqU2dlEEO+8EpPX7K1tkmv7V49y8CFWS7OzDYY/hBzBlpKJ8KYY6/ykiJlBTEPfYwvKRF3Tdh3SJIjhisAEVia3ka7jphe2tgsu0Rl6Djvl+gPxGCIdHk9YRowwWS6N3TsHid9tQN11/lSJ2PzSNQSRlAG0xKHxBleIQzZRekjvMdAlDfDNCEPM/viwgIdShpJxjlG+iv8uoaTtiRFGfi3AZx4N6a8dSji/9WyJUKtOhGGJLx2kKTZ2Syu8uhwrjJJz5HN47u7D/Ox3In6qgJpcLnYNX5x18tyxBimQ6vQa6LtCzpM92tO4kOnxnI8c=
2021-04-23T22:01:55.619 INFO:teuthology.kill:  smithi198.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaeNR48KC4TcQ2aCRNowcUVbFtjyuDBBHYur6XR/BHZccqju1W7805xid2kipJvY+ZW8W6jxVjeOFkDyEwrrVqazljjr1PtrkprI93PqnTPEoSO77LCNP/vxtobhjAM9sxxoDG8UaOdSL73QCNzmirV+t4MZcc5h+H4AHbhlOl5v+21i3DabG3B9F2g4eBoPgnQyIEVdM4IWAUa6zeK8nO5w044TISv1sB7ciVqmhmQieuD97Uliwqbzkgj4w5GTc2kJ606a57GjIUhZfXCDqjtbicZPDYGKxTOthjFZiWy4BOPJKPjVlawOKDE9Zhsr9WoGl0FsPVKNDlMP6hQ4KqmsS9b7KN70SJkX5nYBSuCMtafhmrVD46HaGXpQNfiISxdz3w/xSGiW3Te9HdVorpqLf0ODM9I15VyyTRpcdu1CL1IhM6+cJAM02VNNGij82XAngQ4u5oqm/Hy6fkKnDhUxjzPms8mea1xJj7Mc9/1ZRjPufIsOXXfIQkoh5zn2s=
2021-04-23T22:01:55.641 INFO:teuthology.kill:2021-04-23 22:01:55,641.641 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-23T22:01:55.654 INFO:teuthology.kill:2021-04-23 22:01:55,653.653 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-04-23T22:01:55.726 INFO:teuthology.kill:2021-04-23 22:01:55,725.725 INFO:teuthology.orchestra.console:Power off smithi142
2021-04-23T22:01:55.749 INFO:teuthology.kill:2021-04-23 22:01:55,749.749 INFO:teuthology.orchestra.console:Power off smithi198
2021-04-23T22:01:55.893 INFO:teuthology.kill:2021-04-23 22:01:55,893.893 INFO:teuthology.orchestra.console:Power off for smithi142 completed
2021-04-23T22:01:55.910 INFO:teuthology.kill:2021-04-23 22:01:55,909.909 INFO:teuthology.orchestra.console:Power off for smithi198 completed
2021-04-23T22:01:56.054 INFO:teuthology.kill:2021-04-23 22:01:56,053.053 INFO:teuthology.lock.ops:unlocked smithi142.front.sepia.ceph.com
2021-04-23T22:01:56.072 INFO:teuthology.kill:2021-04-23 22:01:56,071.071 INFO:teuthology.lock.ops:unlocked smithi198.front.sepia.ceph.com
2021-04-23T22:03:56.430 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-04-23T22:03:56.521 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2021-04-23T22:03:56.521 INFO:teuthology.nuke:Checking targets against current locks
2021-04-23T22:03:56.591 INFO:teuthology.nuke:Not nuking smithi142.front.sepia.ceph.com because description doesn't match
2021-04-23T22:03:56.592 INFO:teuthology.nuke:Not nuking smithi198.front.sepia.ceph.com because description doesn't match