2021-05-28T01:12:47.951 INFO:root:teuthology version: 1.1.0-f359b10d 2021-05-28T01:12:48.070 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi013.front.sepia.ceph.com' reimaging 2021-05-28T01:12:48.071 INFO:teuthology.lock.ops:Updating [ubuntu@smithi013.front.sepia.ceph.com]: reset os type and version on server 2021-05-28T01:12:48.072 INFO:teuthology.lock.ops:Updating smithi013.front.sepia.ceph.com on lock server 2021-05-28T01:12:48.090 INFO:teuthology.lock.ops:Node 'ubuntu@smithi013.front.sepia.ceph.com' reimaging is complete 2021-05-28T01:12:48.093 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi081.front.sepia.ceph.com' reimaging 2021-05-28T01:12:48.093 INFO:teuthology.lock.ops:Updating [ubuntu@smithi081.front.sepia.ceph.com]: reset os type and version on server 2021-05-28T01:12:48.094 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2021-05-28T01:12:48.113 INFO:teuthology.lock.ops:Node 'ubuntu@smithi081.front.sepia.ceph.com' reimaging is complete 2021-05-28T01:12:48.304 INFO:teuthology.provision.fog.smithi013:Scheduling deploy of ubuntu 20.04 2021-05-28T01:12:48.318 INFO:teuthology.provision.fog.smithi081:Scheduling deploy of ubuntu 20.04 2021-05-28T01:12:48.668 INFO:teuthology.orchestra.console:Power off smithi013 2021-05-28T01:12:48.710 INFO:teuthology.orchestra.console:Power off smithi081 2021-05-28T01:12:57.048 INFO:teuthology.orchestra.console:Power off for smithi013 completed 2021-05-28T01:12:57.150 INFO:teuthology.orchestra.console:Power on smithi013 2021-05-28T01:13:01.190 INFO:teuthology.orchestra.console:Power off for smithi081 completed 2021-05-28T01:13:01.294 INFO:teuthology.orchestra.console:Power on smithi081 2021-05-28T01:13:05.517 INFO:teuthology.orchestra.console:Power on for smithi013 completed 2021-05-28T01:13:05.545 INFO:teuthology.orchestra.console:Power on for smithi081 completed 2021-05-28T01:13:05.619 INFO:teuthology.provision.fog.smithi013:Waiting for deploy to finish 2021-05-28T01:13:05.646 INFO:teuthology.provision.fog.smithi081:Waiting for deploy to finish 2021-05-28T01:17:34.261 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-28T01:17:34.732 INFO:teuthology.provision.fog.smithi013:Node is ready 2021-05-28T01:17:34.739 INFO:teuthology.orchestra.run.smithi013.stdout:smithi013.front.sepia.ceph.com 2021-05-28T01:17:34.748 INFO:teuthology.orchestra.run.smithi013.stdout:172.21.15.13 smithi013.front.sepia.ceph.com smithi013 2021-05-28T01:17:34.934 INFO:teuthology.provision.fog.smithi013:Deploy complete! 2021-05-28T01:18:02.011 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-28T01:18:02.468 INFO:teuthology.provision.fog.smithi081:Node is ready 2021-05-28T01:18:02.476 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081.front.sepia.ceph.com 2021-05-28T01:18:02.485 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81 smithi081.front.sepia.ceph.com smithi081 2021-05-28T01:18:02.760 INFO:teuthology.provision.fog.smithi081:Deploy complete! 2021-05-28T01:18:02.947 INFO:teuthology.lock.ops:Checking smithi013.front.sepia.ceph.com 2021-05-28T01:18:02.948 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-28T01:18:02.964 INFO:teuthology.lock.ops:Checking smithi081.front.sepia.ceph.com 2021-05-28T01:18:02.964 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-28T01:18:02.979 INFO:teuthology.lock.ops:Updating [smithi013.front.sepia.ceph.com]: set os type and version on server 2021-05-28T01:18:02.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-28T01:18:03.263 INFO:teuthology.orchestra.run.smithi013.stdout:x86_64 2021-05-28T01:18:03.311 INFO:teuthology.orchestra.run.smithi013.stdout:NAME="Ubuntu" 2021-05-28T01:18:03.311 INFO:teuthology.orchestra.run.smithi013.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-05-28T01:18:03.312 INFO:teuthology.orchestra.run.smithi013.stdout:ID=ubuntu 2021-05-28T01:18:03.312 INFO:teuthology.orchestra.run.smithi013.stdout:ID_LIKE=debian 2021-05-28T01:18:03.312 INFO:teuthology.orchestra.run.smithi013.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-05-28T01:18:03.312 INFO:teuthology.orchestra.run.smithi013.stdout:VERSION_ID="20.04" 2021-05-28T01:18:03.313 INFO:teuthology.orchestra.run.smithi013.stdout:HOME_URL="https://www.ubuntu.com/" 2021-05-28T01:18:03.313 INFO:teuthology.orchestra.run.smithi013.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-05-28T01:18:03.313 INFO:teuthology.orchestra.run.smithi013.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-05-28T01:18:03.313 INFO:teuthology.orchestra.run.smithi013.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-05-28T01:18:03.314 INFO:teuthology.orchestra.run.smithi013.stdout:VERSION_CODENAME=focal 2021-05-28T01:18:03.314 INFO:teuthology.orchestra.run.smithi013.stdout:UBUNTU_CODENAME=focal 2021-05-28T01:18:03.315 INFO:teuthology.lock.ops:Updating smithi013.front.sepia.ceph.com on lock server 2021-05-28T01:18:03.333 INFO:teuthology.lock.ops:Updating [smithi081.front.sepia.ceph.com]: set os type and version on server 2021-05-28T01:18:03.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-28T01:18:03.502 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2021-05-28T01:18:03.550 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="Ubuntu" 2021-05-28T01:18:03.551 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-05-28T01:18:03.551 INFO:teuthology.orchestra.run.smithi081.stdout:ID=ubuntu 2021-05-28T01:18:03.551 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE=debian 2021-05-28T01:18:03.551 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-05-28T01:18:03.552 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="20.04" 2021-05-28T01:18:03.552 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://www.ubuntu.com/" 2021-05-28T01:18:03.552 INFO:teuthology.orchestra.run.smithi081.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-05-28T01:18:03.552 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-05-28T01:18:03.552 INFO:teuthology.orchestra.run.smithi081.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-05-28T01:18:03.553 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_CODENAME=focal 2021-05-28T01:18:03.553 INFO:teuthology.orchestra.run.smithi081.stdout:UBUNTU_CODENAME=focal 2021-05-28T01:18:03.553 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2021-05-28T01:18:03.634 INFO:teuthology.dispatcher.supervisor:Running job 6139077 2021-05-28T01:18:03.634 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/virtualenv/bin/teuthology -v --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2021-05-27_23:16:20-rados-wip-sage3-testing-2021-05-25-1649-distro-basic-smithi/6139077 --name sage-2021-05-27_23:16:20-rados-wip-sage3-testing-2021-05-25-1649-distro-basic-smithi --description rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/snaps-few-objects} -- /home/teuthworker/archive/sage-2021-05-27_23:16:20-rados-wip-sage3-testing-2021-05-25-1649-distro-basic-smithi/6139077/orig.config.yaml 2021-05-28T01:18:03.640 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/sage-2021-05-27_23:16:20-rados-wip-sage3-testing-2021-05-25-1649-distro-basic-smithi/6139077 2021-05-28T01:18:03.641 INFO:teuthology.dispatcher.supervisor:Job PID: 21075 2021-05-28T01:18:03.642 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-28T13:18:55.279 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-28T13:18:55.527 INFO:teuthology.kill:Killing Pids: {21075} 2021-05-28T13:18:56.662 INFO:teuthology.kill:Nuking machines: ['smithi013', 'smithi081'] 2021-05-28T13:18:57.583 INFO:teuthology.kill:2021-05-28 13:18:57,582.582 INFO:teuthology.nuke:targets: 2021-05-28T13:18:57.584 INFO:teuthology.kill: smithi013.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCTUCcHgJkdVj0gQCajZEjjXWy4eOTlFj8oFmBtW/liEQjmyi7zqZ3SxT7yqbauolV7E8+u98Efk5DUErwywMlwl6vTw3p6juxbdjS0Cz+1INtMBUqFN8NUjFjzxSJw+BN4KgFAeanqVMGqPTXvUxGUAQw701TaEJTDOSqbVEntdOVly7dbsYEMmYMXrBKybrTwFW/2C2QPansOkekL536aAN8ilB31JgcfptXK4Ln3cGASbH4tZg1pdbtRdSRU2AHJ9xlYg6yhTlDAXlSlBHdxf5n3LGB4tQsMY1uvO1b6PLhbsn/eOdH7QciTw6g9QNQmlDQUn2Yw+EgzGEX5WnQpOGq/5/LCV/HLBxDsQLTqY5xM7I73/ZtElOThLQc8Ci6S0tnCc4ZGQtpt5Cu4tMdiaYWxuvoe2IgmCnS8q+bU18OUu3/FQSiSg3WpSjxLQpd4n5PsNwPwCqNQo9yP0o0riXx5pgXeIS10oMGCPBV6PTR+H98oDFSIiLaf28ENUMc= 2021-05-28T13:18:57.584 INFO:teuthology.kill: smithi081.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCsqP9GZ7LzxzwEr9bVMwGSyoQA7HxCTjtP3GLSbCwZhIj+AfVQaoMZCX8xbLjvSmGBD6MtYtrisWdBCR33MEu53xwwdIHVRAHs55jyvgM06wC3dpIM8mF1qmPZXhv+ROZabuQHw3bcI7CpNDOME3ZvgXgVgWHscKKZZbTOksoBT/xoNEE0RZgKuMDyT4MeeDmL2idbXmXIIQY+JuFSScpDOrgpw9v5Ispky52dHe6LqJ0OCdJkV5S4aZUVTdzcd3pb4gY7RpsC304zeRMPwn84CaT5SLC9Znr0zTbxVy20jMh5+EhXtAxkSpA3R2aETXaYBPwNJ/7Rh12wiPPJpLTqOf9NbzKs14bTQZ/70VXfqu1Z36YUm75xewbKn3c4ghluVaHlAMkFNhrwJFlve1YF9jZE6QZmVyTteJgX4r/qjA21N2XzdUW+MVax4YO1LQQK3LZHxKzqkXOmnws+60DbfaADmb/Y3X8rNrKwwaBpJD5Jc2ulcbRzOJFJruTLOq8= 2021-05-28T13:18:57.584 INFO:teuthology.kill:2021-05-28 13:18:57,583.583 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-28T13:18:57.585 INFO:teuthology.kill:2021-05-28 13:18:57,585.585 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-28T13:18:57.647 INFO:teuthology.kill:2021-05-28 13:18:57,647.647 INFO:teuthology.orchestra.console:Power off smithi013 2021-05-28T13:18:57.661 INFO:teuthology.kill:2021-05-28 13:18:57,661.661 INFO:teuthology.orchestra.console:Power off smithi081 2021-05-28T13:19:10.125 INFO:teuthology.kill:2021-05-28 13:19:10,125.125 INFO:teuthology.orchestra.console:Power off for smithi013 completed 2021-05-28T13:19:10.131 INFO:teuthology.kill:2021-05-28 13:19:10,131.131 INFO:teuthology.orchestra.console:Power off for smithi081 completed 2021-05-28T13:19:10.350 INFO:teuthology.task.internal:roles: ubuntu@smithi013.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'] 2021-05-28T13:19:10.350 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2021-05-28T13:19:10.351 INFO:teuthology.misc:Compressing logs... 2021-05-28T13:19:10.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-28T13:20:10.415 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-28T13:20:10.475 INFO:teuthology.kill:No teuthology processes running 2021-05-28T13:20:11.608 INFO:teuthology.kill:Nuking machines: ['smithi013', 'smithi081'] 2021-05-28T13:20:12.526 INFO:teuthology.kill:2021-05-28 13:20:12,526.526 INFO:teuthology.nuke:targets: 2021-05-28T13:20:12.527 INFO:teuthology.kill: smithi013.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCTUCcHgJkdVj0gQCajZEjjXWy4eOTlFj8oFmBtW/liEQjmyi7zqZ3SxT7yqbauolV7E8+u98Efk5DUErwywMlwl6vTw3p6juxbdjS0Cz+1INtMBUqFN8NUjFjzxSJw+BN4KgFAeanqVMGqPTXvUxGUAQw701TaEJTDOSqbVEntdOVly7dbsYEMmYMXrBKybrTwFW/2C2QPansOkekL536aAN8ilB31JgcfptXK4Ln3cGASbH4tZg1pdbtRdSRU2AHJ9xlYg6yhTlDAXlSlBHdxf5n3LGB4tQsMY1uvO1b6PLhbsn/eOdH7QciTw6g9QNQmlDQUn2Yw+EgzGEX5WnQpOGq/5/LCV/HLBxDsQLTqY5xM7I73/ZtElOThLQc8Ci6S0tnCc4ZGQtpt5Cu4tMdiaYWxuvoe2IgmCnS8q+bU18OUu3/FQSiSg3WpSjxLQpd4n5PsNwPwCqNQo9yP0o0riXx5pgXeIS10oMGCPBV6PTR+H98oDFSIiLaf28ENUMc= 2021-05-28T13:20:12.528 INFO:teuthology.kill: smithi081.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCsqP9GZ7LzxzwEr9bVMwGSyoQA7HxCTjtP3GLSbCwZhIj+AfVQaoMZCX8xbLjvSmGBD6MtYtrisWdBCR33MEu53xwwdIHVRAHs55jyvgM06wC3dpIM8mF1qmPZXhv+ROZabuQHw3bcI7CpNDOME3ZvgXgVgWHscKKZZbTOksoBT/xoNEE0RZgKuMDyT4MeeDmL2idbXmXIIQY+JuFSScpDOrgpw9v5Ispky52dHe6LqJ0OCdJkV5S4aZUVTdzcd3pb4gY7RpsC304zeRMPwn84CaT5SLC9Znr0zTbxVy20jMh5+EhXtAxkSpA3R2aETXaYBPwNJ/7Rh12wiPPJpLTqOf9NbzKs14bTQZ/70VXfqu1Z36YUm75xewbKn3c4ghluVaHlAMkFNhrwJFlve1YF9jZE6QZmVyTteJgX4r/qjA21N2XzdUW+MVax4YO1LQQK3LZHxKzqkXOmnws+60DbfaADmb/Y3X8rNrKwwaBpJD5Jc2ulcbRzOJFJruTLOq8= 2021-05-28T13:20:12.547 INFO:teuthology.kill:2021-05-28 13:20:12,546.546 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-28T13:20:12.549 INFO:teuthology.kill:2021-05-28 13:20:12,549.549 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-28T13:20:12.603 INFO:teuthology.kill:2021-05-28 13:20:12,603.603 INFO:teuthology.orchestra.console:Power off smithi013 2021-05-28T13:20:12.617 INFO:teuthology.kill:2021-05-28 13:20:12,616.616 INFO:teuthology.orchestra.console:Power off smithi081 2021-05-28T13:20:12.746 INFO:teuthology.kill:2021-05-28 13:20:12,745.745 INFO:teuthology.orchestra.console:Power off for smithi013 completed 2021-05-28T13:20:12.759 INFO:teuthology.kill:2021-05-28 13:20:12,759.759 INFO:teuthology.orchestra.console:Power off for smithi081 completed 2021-05-28T13:20:12.887 INFO:teuthology.kill:2021-05-28 13:20:12,886.886 INFO:teuthology.lock.ops:unlocked smithi013.front.sepia.ceph.com 2021-05-28T13:20:12.904 INFO:teuthology.kill:2021-05-28 13:20:12,904.904 INFO:teuthology.lock.ops:unlocked smithi081.front.sepia.ceph.com 2021-05-28T13:22:13.196 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-05-28T13:22:13.278 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-05-28T13:22:13.278 INFO:teuthology.nuke:Checking targets against current locks 2021-05-28T13:22:13.349 INFO:teuthology.nuke:Not nuking smithi013.front.sepia.ceph.com because description doesn't match 2021-05-28T13:22:13.350 INFO:teuthology.nuke:Not nuking smithi081.front.sepia.ceph.com because description doesn't match