2021-04-23T09:50:19.951 INFO:root:teuthology version: 1.1.0-2713a3cd 2021-04-23T09:50:20.059 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi028.front.sepia.ceph.com' reimaging 2021-04-23T09:50:20.059 INFO:teuthology.lock.ops:Updating [ubuntu@smithi028.front.sepia.ceph.com]: reset os type and version on server 2021-04-23T09:50:20.059 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2021-04-23T09:50:20.074 INFO:teuthology.lock.ops:Node 'ubuntu@smithi028.front.sepia.ceph.com' reimaging is complete 2021-04-23T09:50:20.075 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi170.front.sepia.ceph.com' reimaging 2021-04-23T09:50:20.075 INFO:teuthology.lock.ops:Updating [ubuntu@smithi170.front.sepia.ceph.com]: reset os type and version on server 2021-04-23T09:50:20.075 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2021-04-23T09:50:20.092 INFO:teuthology.lock.ops:Node 'ubuntu@smithi170.front.sepia.ceph.com' reimaging is complete 2021-04-23T09:50:20.319 INFO:teuthology.provision.fog.smithi028:Scheduling deploy of ubuntu 20.04 2021-04-23T09:50:20.397 INFO:teuthology.provision.fog.smithi170:Scheduling deploy of ubuntu 20.04 2021-04-23T09:50:20.804 INFO:teuthology.orchestra.console:Power off smithi028 2021-04-23T09:50:20.893 INFO:teuthology.orchestra.console:Power off smithi170 2021-04-23T09:50:20.949 INFO:teuthology.orchestra.console:Power off for smithi028 completed 2021-04-23T09:50:21.032 INFO:teuthology.orchestra.console:Power off for smithi170 completed 2021-04-23T09:50:21.050 INFO:teuthology.orchestra.console:Power on smithi028 2021-04-23T09:50:21.133 INFO:teuthology.orchestra.console:Power on smithi170 2021-04-23T09:50:29.417 INFO:teuthology.orchestra.console:Power on for smithi028 completed 2021-04-23T09:50:29.497 INFO:teuthology.orchestra.console:Power on for smithi170 completed 2021-04-23T09:50:29.518 INFO:teuthology.provision.fog.smithi028:Waiting for deploy to finish 2021-04-23T09:50:29.598 INFO:teuthology.provision.fog.smithi170:Waiting for deploy to finish 2021-04-23T09:54:58.275 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-23T09:54:58.737 INFO:teuthology.provision.fog.smithi028:Node is ready 2021-04-23T09:54:58.743 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028.front.sepia.ceph.com 2021-04-23T09:54:58.793 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28 smithi028.front.sepia.ceph.com smithi028 2021-04-23T09:54:58.992 INFO:teuthology.provision.fog.smithi028:Deploy complete! 2021-04-23T09:55:25.126 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-23T09:55:25.560 INFO:teuthology.provision.fog.smithi170:Node is ready 2021-04-23T09:55:25.566 INFO:teuthology.orchestra.run.smithi170.stdout:smithi170.front.sepia.ceph.com 2021-04-23T09:55:25.615 INFO:teuthology.orchestra.run.smithi170.stdout:172.21.15.170 smithi170.front.sepia.ceph.com smithi170 2021-04-23T09:55:25.821 INFO:teuthology.provision.fog.smithi170:Deploy complete! 2021-04-23T09:55:26.471 INFO:teuthology.lock.ops:Checking smithi028.front.sepia.ceph.com 2021-04-23T09:55:26.472 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-23T09:55:26.490 INFO:teuthology.lock.ops:Checking smithi170.front.sepia.ceph.com 2021-04-23T09:55:26.490 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-23T09:55:26.668 INFO:teuthology.lock.ops:Updating [smithi028.front.sepia.ceph.com]: set os type and version on server 2021-04-23T09:55:26.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-23T09:55:26.980 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2021-04-23T09:55:26.986 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="Ubuntu" 2021-04-23T09:55:26.986 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-04-23T09:55:26.986 INFO:teuthology.orchestra.run.smithi028.stdout:ID=ubuntu 2021-04-23T09:55:26.987 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE=debian 2021-04-23T09:55:26.987 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-04-23T09:55:26.987 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="20.04" 2021-04-23T09:55:26.987 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://www.ubuntu.com/" 2021-04-23T09:55:26.987 INFO:teuthology.orchestra.run.smithi028.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-04-23T09:55:26.987 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-04-23T09:55:26.987 INFO:teuthology.orchestra.run.smithi028.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-04-23T09:55:26.987 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_CODENAME=focal 2021-04-23T09:55:26.987 INFO:teuthology.orchestra.run.smithi028.stdout:UBUNTU_CODENAME=focal 2021-04-23T09:55:26.988 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2021-04-23T09:55:27.004 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: set os type and version on server 2021-04-23T09:55:27.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-23T09:55:27.175 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2021-04-23T09:55:27.222 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="Ubuntu" 2021-04-23T09:55:27.222 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-04-23T09:55:27.223 INFO:teuthology.orchestra.run.smithi170.stdout:ID=ubuntu 2021-04-23T09:55:27.223 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE=debian 2021-04-23T09:55:27.223 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-04-23T09:55:27.223 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="20.04" 2021-04-23T09:55:27.223 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://www.ubuntu.com/" 2021-04-23T09:55:27.223 INFO:teuthology.orchestra.run.smithi170.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-04-23T09:55:27.223 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-04-23T09:55:27.223 INFO:teuthology.orchestra.run.smithi170.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-04-23T09:55:27.223 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_CODENAME=focal 2021-04-23T09:55:27.223 INFO:teuthology.orchestra.run.smithi170.stdout:UBUNTU_CODENAME=focal 2021-04-23T09:55:27.224 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2021-04-23T09:55:27.370 INFO:teuthology.dispatcher.supervisor:Running job 6064373 2021-04-23T09:55:27.370 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2021-04-22_11:34:56-rados-no_column_b-distro-basic-smithi/6064373 --name benhanokh-2021-04-22_11:34:56-rados-no_column_b-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/small-objects-balanced} -- /home/teuthworker/archive/benhanokh-2021-04-22_11:34:56-rados-no_column_b-distro-basic-smithi/6064373/orig.config.yaml 2021-04-23T09:55:27.373 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2021-04-22_11:34:56-rados-no_column_b-distro-basic-smithi/6064373 2021-04-23T09:55:27.373 INFO:teuthology.dispatcher.supervisor:Job PID: 16853 2021-04-23T09:55:27.373 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-04-23T21:57:02.872 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-04-23T21:57:03.085 INFO:teuthology.kill:Killing Pids: {16853} 2021-04-23T21:57:04.161 INFO:teuthology.kill:Nuking machines: ['smithi028', 'smithi170'] 2021-04-23T21:57:05.093 INFO:teuthology.kill:2021-04-23 21:57:05,092.092 INFO:teuthology.nuke:targets: 2021-04-23T21:57:05.093 INFO:teuthology.kill: smithi028.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8i4eystlOYXyoupec21h8V0uNXBcPSxv82H8Cb9w9F/ObPBI84FlbiuJwRaEHLwfSdX5cNTWNuUqQiR/tQiNtXcR86IWqLT/cRuer4yg2mj3pgxpIGOJ5LrwpnwosdZxPgXBUrTkSEeF+9f55QMoxdqVqjZ2gRYCf4fyFVmhsW07Y0DGLHs8j+Vu7EcXBAY2v3ElSc5WIgaXf8QN0V23zPbd8Bla1ByGVLpe5mVaCs+1iej98tTL9L2vE6VcHg/ud7nF6yhpT5Q52t9AHgCQWHbnK4r4WHrWf5SkRkM2+bqC6ZU8u/9rsu8+nyiaO1tCXBNPbVmkGKFmar2hMn1XqDbu7IuePxy7keibsoKuCy485QqrvUoulabxqPV6X4yNJreeh7O/VM6APs5S3EPbIsX7CSNINNgE+xH6riOxFYsqRYiMzuOnCiWWCor1ZaTIxDbWzaUGh2DPBePY4b+RGZ/smgYP/4OlQb1+XBuAR+IKT+L9k+TcM0zWGixgFoJU= 2021-04-23T21:57:05.093 INFO:teuthology.kill: smithi170.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZYuayO98ONeZh9pD3QNsSlzB1d5fChw7zX8R77F95KiMaLjT+HVQVnPFX/oQdfF6i1i5y/5Go4h2LDVujq8JCrugdctO8mhsmyeR5CRFW9wnY3eLePt1BWkCO2d+EFfMvYKKXDEP5b/uLItoO+ekI8ektO7JFUl+pzSx18CjHzvGesk6RnQC0Dk7HwGB2icu1CyAi0jCfOiu9IgjNS5tsOgu+3HyAgwNcJVYJ2YriJI+J5PgkDzpLswXa90ILitWR/8yTscCFmCwOh22Td4RgnThmgncgVxL/YtmXP+4nKKio4srQEYQ4Q0rbz2ivL+f3Dz9t6y8RUJaSwDBBJsRN+QF1TYKolf7KToTuUm/YTD2emk/de9aS0WxntMIvMQOOk0J6VzJ+qsi6qT49SwDakQ/jY2YfH45CWdCzO4i3ZYgmnaxZpautseGDcVX1fm2OMTC0NV7i2YliblDbAIpyiKQWk7+4/LKrnnsGhAUEUAbqxV71yUYWo+yTuipq39M= 2021-04-23T21:57:05.093 INFO:teuthology.kill:2021-04-23 21:57:05,093.093 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-23T21:57:05.095 INFO:teuthology.kill:2021-04-23 21:57:05,095.095 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-23T21:57:05.170 INFO:teuthology.kill:2021-04-23 21:57:05,170.170 INFO:teuthology.orchestra.console:Power off smithi028 2021-04-23T21:57:05.184 INFO:teuthology.kill:2021-04-23 21:57:05,184.184 INFO:teuthology.orchestra.console:Power off smithi170 2021-04-23T21:57:13.554 INFO:teuthology.kill:2021-04-23 21:57:13,553.553 INFO:teuthology.orchestra.console:Power off for smithi028 completed 2021-04-23T21:57:17.662 INFO:teuthology.kill:2021-04-23 21:57:17,662.662 INFO:teuthology.orchestra.console:Power off for smithi170 completed 2021-04-23T21:57:17.930 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-04-23T21:57:17.930 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-04-23T21:57:17.930 INFO:teuthology.misc:Compressing logs... 2021-04-23T21:57:17.930 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-23T21:57:52.608 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-04-23T21:57:52.671 INFO:teuthology.kill:No teuthology processes running 2021-04-23T21:57:53.701 INFO:teuthology.kill:Nuking machines: ['smithi028', 'smithi170'] 2021-04-23T21:57:54.638 INFO:teuthology.kill:2021-04-23 21:57:54,638.638 INFO:teuthology.nuke:targets: 2021-04-23T21:57:54.639 INFO:teuthology.kill: smithi028.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8i4eystlOYXyoupec21h8V0uNXBcPSxv82H8Cb9w9F/ObPBI84FlbiuJwRaEHLwfSdX5cNTWNuUqQiR/tQiNtXcR86IWqLT/cRuer4yg2mj3pgxpIGOJ5LrwpnwosdZxPgXBUrTkSEeF+9f55QMoxdqVqjZ2gRYCf4fyFVmhsW07Y0DGLHs8j+Vu7EcXBAY2v3ElSc5WIgaXf8QN0V23zPbd8Bla1ByGVLpe5mVaCs+1iej98tTL9L2vE6VcHg/ud7nF6yhpT5Q52t9AHgCQWHbnK4r4WHrWf5SkRkM2+bqC6ZU8u/9rsu8+nyiaO1tCXBNPbVmkGKFmar2hMn1XqDbu7IuePxy7keibsoKuCy485QqrvUoulabxqPV6X4yNJreeh7O/VM6APs5S3EPbIsX7CSNINNgE+xH6riOxFYsqRYiMzuOnCiWWCor1ZaTIxDbWzaUGh2DPBePY4b+RGZ/smgYP/4OlQb1+XBuAR+IKT+L9k+TcM0zWGixgFoJU= 2021-04-23T21:57:54.639 INFO:teuthology.kill: smithi170.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZYuayO98ONeZh9pD3QNsSlzB1d5fChw7zX8R77F95KiMaLjT+HVQVnPFX/oQdfF6i1i5y/5Go4h2LDVujq8JCrugdctO8mhsmyeR5CRFW9wnY3eLePt1BWkCO2d+EFfMvYKKXDEP5b/uLItoO+ekI8ektO7JFUl+pzSx18CjHzvGesk6RnQC0Dk7HwGB2icu1CyAi0jCfOiu9IgjNS5tsOgu+3HyAgwNcJVYJ2YriJI+J5PgkDzpLswXa90ILitWR/8yTscCFmCwOh22Td4RgnThmgncgVxL/YtmXP+4nKKio4srQEYQ4Q0rbz2ivL+f3Dz9t6y8RUJaSwDBBJsRN+QF1TYKolf7KToTuUm/YTD2emk/de9aS0WxntMIvMQOOk0J6VzJ+qsi6qT49SwDakQ/jY2YfH45CWdCzO4i3ZYgmnaxZpautseGDcVX1fm2OMTC0NV7i2YliblDbAIpyiKQWk7+4/LKrnnsGhAUEUAbqxV71yUYWo+yTuipq39M= 2021-04-23T21:57:54.666 INFO:teuthology.kill:2021-04-23 21:57:54,666.666 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-23T21:57:54.684 INFO:teuthology.kill:2021-04-23 21:57:54,683.683 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-23T21:57:54.756 INFO:teuthology.kill:2021-04-23 21:57:54,756.756 INFO:teuthology.orchestra.console:Power off smithi028 2021-04-23T21:57:54.769 INFO:teuthology.kill:2021-04-23 21:57:54,769.769 INFO:teuthology.orchestra.console:Power off smithi170 2021-04-23T21:57:54.900 INFO:teuthology.kill:2021-04-23 21:57:54,899.899 INFO:teuthology.orchestra.console:Power off for smithi028 completed 2021-04-23T21:57:54.913 INFO:teuthology.kill:2021-04-23 21:57:54,912.912 INFO:teuthology.orchestra.console:Power off for smithi170 completed 2021-04-23T21:57:55.060 INFO:teuthology.kill:2021-04-23 21:57:55,060.060 INFO:teuthology.lock.ops:unlocked smithi028.front.sepia.ceph.com 2021-04-23T21:57:55.078 INFO:teuthology.kill:2021-04-23 21:57:55,078.078 INFO:teuthology.lock.ops:unlocked smithi170.front.sepia.ceph.com 2021-04-23T22:00:09.984 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-04-23T22:00:10.111 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-04-23T22:00:10.111 INFO:teuthology.nuke:Checking targets against current locks 2021-04-23T22:00:10.256 INFO:teuthology.nuke:Not nuking smithi170.front.sepia.ceph.com because description doesn't match 2021-04-23T22:00:10.257 INFO:teuthology.nuke:Not nuking smithi028.front.sepia.ceph.com because description doesn't match