2021-04-23T12:02:20.678 INFO:root:teuthology version: 1.1.0-2713a3cd 2021-04-23T12:02:20.793 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi106.front.sepia.ceph.com' reimaging 2021-04-23T12:02:20.793 INFO:teuthology.lock.ops:Updating [ubuntu@smithi106.front.sepia.ceph.com]: reset os type and version on server 2021-04-23T12:02:20.793 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server 2021-04-23T12:02:20.809 INFO:teuthology.lock.ops:Node 'ubuntu@smithi106.front.sepia.ceph.com' reimaging is complete 2021-04-23T12:02:20.809 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi145.front.sepia.ceph.com' reimaging 2021-04-23T12:02:20.809 INFO:teuthology.lock.ops:Updating [ubuntu@smithi145.front.sepia.ceph.com]: reset os type and version on server 2021-04-23T12:02:20.809 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2021-04-23T12:02:20.824 INFO:teuthology.lock.ops:Node 'ubuntu@smithi145.front.sepia.ceph.com' reimaging is complete 2021-04-23T12:02:21.085 INFO:teuthology.provision.fog.smithi106:Scheduling deploy of ubuntu 20.04 2021-04-23T12:02:21.100 INFO:teuthology.provision.fog.smithi145:Scheduling deploy of ubuntu 20.04 2021-04-23T12:02:21.491 INFO:teuthology.orchestra.console:Power off smithi106 2021-04-23T12:02:21.525 INFO:teuthology.orchestra.console:Power off smithi145 2021-04-23T12:02:21.633 INFO:teuthology.orchestra.console:Power off for smithi106 completed 2021-04-23T12:02:21.664 INFO:teuthology.orchestra.console:Power off for smithi145 completed 2021-04-23T12:02:21.734 INFO:teuthology.orchestra.console:Power on smithi106 2021-04-23T12:02:21.765 INFO:teuthology.orchestra.console:Power on smithi145 2021-04-23T12:02:30.096 INFO:teuthology.orchestra.console:Power on for smithi106 completed 2021-04-23T12:02:30.144 INFO:teuthology.orchestra.console:Power on for smithi145 completed 2021-04-23T12:02:30.197 INFO:teuthology.provision.fog.smithi106:Waiting for deploy to finish 2021-04-23T12:02:30.245 INFO:teuthology.provision.fog.smithi145:Waiting for deploy to finish 2021-04-23T12:05:16.538 ERROR:teuthology.orchestra.connection:Error authenticating with smithi106.front.sepia.ceph.com: Authentication failed. 2021-04-23T12:07:20.248 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-23T12:07:20.682 INFO:teuthology.provision.fog.smithi106:Node is ready 2021-04-23T12:07:20.688 INFO:teuthology.orchestra.run.smithi106.stdout:smithi106.front.sepia.ceph.com 2021-04-23T12:07:20.741 INFO:teuthology.orchestra.run.smithi106.stdout:172.21.15.106 smithi106.front.sepia.ceph.com smithi106 2021-04-23T12:07:20.930 INFO:teuthology.provision.fog.smithi106:Deploy complete! 2021-04-23T12:07:30.556 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-23T12:07:31.032 INFO:teuthology.provision.fog.smithi145:Node is ready 2021-04-23T12:07:31.079 INFO:teuthology.orchestra.run.smithi145.stdout:smithi145.front.sepia.ceph.com 2021-04-23T12:07:31.086 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145 smithi145.front.sepia.ceph.com smithi145 2021-04-23T12:07:31.258 INFO:teuthology.provision.fog.smithi145:Deploy complete! 2021-04-23T12:07:31.407 INFO:teuthology.lock.ops:Checking smithi106.front.sepia.ceph.com 2021-04-23T12:07:31.407 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-23T12:07:31.423 INFO:teuthology.lock.ops:Checking smithi145.front.sepia.ceph.com 2021-04-23T12:07:31.423 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-23T12:07:31.438 INFO:teuthology.lock.ops:Updating [smithi106.front.sepia.ceph.com]: set os type and version on server 2021-04-23T12:07:31.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-23T12:07:31.610 INFO:teuthology.orchestra.run.smithi106.stdout:x86_64 2021-04-23T12:07:31.617 INFO:teuthology.orchestra.run.smithi106.stdout:NAME="Ubuntu" 2021-04-23T12:07:31.617 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-04-23T12:07:31.617 INFO:teuthology.orchestra.run.smithi106.stdout:ID=ubuntu 2021-04-23T12:07:31.617 INFO:teuthology.orchestra.run.smithi106.stdout:ID_LIKE=debian 2021-04-23T12:07:31.617 INFO:teuthology.orchestra.run.smithi106.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-04-23T12:07:31.617 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_ID="20.04" 2021-04-23T12:07:31.617 INFO:teuthology.orchestra.run.smithi106.stdout:HOME_URL="https://www.ubuntu.com/" 2021-04-23T12:07:31.617 INFO:teuthology.orchestra.run.smithi106.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-04-23T12:07:31.618 INFO:teuthology.orchestra.run.smithi106.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-04-23T12:07:31.618 INFO:teuthology.orchestra.run.smithi106.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-04-23T12:07:31.618 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_CODENAME=focal 2021-04-23T12:07:31.618 INFO:teuthology.orchestra.run.smithi106.stdout:UBUNTU_CODENAME=focal 2021-04-23T12:07:31.619 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server 2021-04-23T12:07:31.642 INFO:teuthology.lock.ops:Updating [smithi145.front.sepia.ceph.com]: set os type and version on server 2021-04-23T12:07:31.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-23T12:07:31.814 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2021-04-23T12:07:31.821 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="Ubuntu" 2021-04-23T12:07:31.821 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-04-23T12:07:31.821 INFO:teuthology.orchestra.run.smithi145.stdout:ID=ubuntu 2021-04-23T12:07:31.821 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE=debian 2021-04-23T12:07:31.822 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-04-23T12:07:31.822 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="20.04" 2021-04-23T12:07:31.822 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://www.ubuntu.com/" 2021-04-23T12:07:31.822 INFO:teuthology.orchestra.run.smithi145.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-04-23T12:07:31.822 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-04-23T12:07:31.822 INFO:teuthology.orchestra.run.smithi145.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-04-23T12:07:31.822 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_CODENAME=focal 2021-04-23T12:07:31.822 INFO:teuthology.orchestra.run.smithi145.stdout:UBUNTU_CODENAME=focal 2021-04-23T12:07:31.822 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2021-04-23T12:07:31.899 INFO:teuthology.dispatcher.supervisor:Running job 6064403 2021-04-23T12:07:31.899 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2021-04-22_11:34:56-rados-no_column_b-distro-basic-smithi/6064403 --name benhanokh-2021-04-22_11:34:56-rados-no_column_b-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects-localized} -- /home/teuthworker/archive/benhanokh-2021-04-22_11:34:56-rados-no_column_b-distro-basic-smithi/6064403/orig.config.yaml 2021-04-23T12:07:31.902 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2021-04-22_11:34:56-rados-no_column_b-distro-basic-smithi/6064403 2021-04-23T12:07:31.903 INFO:teuthology.dispatcher.supervisor:Job PID: 25720 2021-04-23T12:07:31.903 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-04-24T00:08:28.242 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-04-24T00:08:28.401 INFO:teuthology.kill:Killing Pids: {25720} 2021-04-24T00:08:29.610 INFO:teuthology.kill:Nuking machines: ['smithi106', 'smithi145'] 2021-04-24T00:08:30.561 INFO:teuthology.kill:2021-04-24 00:08:30,561.561 INFO:teuthology.nuke:targets: 2021-04-24T00:08:30.562 INFO:teuthology.kill: smithi106.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllKHusPCmgueNxgU2psf2lVm07felMo6pzrJK0ORRefek2hd+GtWk+szBVnERUeaiUKG+Adt0bdCfnO0KeuB37DGmTrC9E7QGndtCRavas8L7tPj1oYFRuXmBqAv/8Z+NBL5ARyofTV8w7CuZiraaWvqeNbpui8j1f2BEigyztomz+J5QJ5MSAFwHa26EG25EorafxjjEr8cnCWA0CsN/gLqj7uQrnUcVcHgmJSZfo2VUIYtus/MJqXJeXxMnXjgqGPia/0zGloPbdiUnDt0t5O3krlFZ9bq3Mf0svNlxmUokZGBywLzpZsvqCHVWpUBHJUb/ADOWt8Sc6/755T6M55hDUDskMtmBGmGob+Xh7KkPUHkuMV9JaF52qh6nNQ3qfjDcbNEsowIPlMewkqyOkBd9tJPqnxHgw+tObArY4Olw8enfuoau2Utk/03gJBPnKxN7uCaE0sRVEzED5Nv/vEd8QHODThhIdIUSf2rhbMyOKW/CwSuiAFUg1rQQ3Tk= 2021-04-24T00:08:30.562 INFO:teuthology.kill: smithi145.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtT/OJXIYBNariV5fKY1PXFx+WXW53P6MDw/c7dKv/U7DZ2TU1B+YE3O8iHbw2gG//9/D3Rlbo9mIIn/8dwvgnXmgh1WGcRl40Ezo0ax+RIEPGQ+DTfTns0M34NtQ5SrpoilJNkog6d82ODjGJRHVVCbGxu2aPx3tQhQByowKFya4LXxN21bNQ0o8KF6ps22joa+Jyrq+qzL9yufeMJ2j1vWLa1IJhdePueUEPnXHdZ2b2QwjksljnvyItabrZg7JbMX9V8/slFTNDo1BkdpVv6XfzdWLE9lWXY1ZePVJxWmSmhhcYH2WlSFnxHbfYd/FaIvv4FcdEaLfO5TrfNr5vqyRWIyttU6OETbUrtzFN0EkuXN0qDSe8Bck9x11liHyttWmEYNkpHZGxV4lXqCVHSYSlufgoGhG0rfKTRXk1LSWx1FC4MGt5Zr+xVOOX8rOmznktWFaaca8v2jL1IyPMroQfbwPZpWBmSVFdUI2k7p9OaoXCEJx4VfnEqtSANLM= 2021-04-24T00:08:30.562 INFO:teuthology.kill:2021-04-24 00:08:30,562.562 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-24T00:08:30.565 INFO:teuthology.kill:2021-04-24 00:08:30,564.564 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-24T00:08:30.658 INFO:teuthology.kill:2021-04-24 00:08:30,657.657 INFO:teuthology.orchestra.console:Power off smithi106 2021-04-24T00:08:30.674 INFO:teuthology.kill:2021-04-24 00:08:30,674.674 INFO:teuthology.orchestra.console:Power off smithi145 2021-04-24T00:08:39.062 INFO:teuthology.kill:2021-04-24 00:08:39,061.061 INFO:teuthology.orchestra.console:Power off for smithi106 completed 2021-04-24T00:08:39.070 INFO:teuthology.kill:2021-04-24 00:08:39,070.070 INFO:teuthology.orchestra.console:Power off for smithi145 completed 2021-04-24T00:08:39.338 INFO:teuthology.task.internal:roles: ubuntu@smithi106.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-04-24T00:08:39.338 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-04-24T00:08:39.338 INFO:teuthology.misc:Compressing logs... 2021-04-24T00:08:39.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-24T00:09:13.820 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-04-24T00:09:13.883 INFO:teuthology.kill:No teuthology processes running 2021-04-24T00:09:15.032 INFO:teuthology.kill:Nuking machines: ['smithi106', 'smithi145'] 2021-04-24T00:09:16.039 INFO:teuthology.kill:2021-04-24 00:09:16,039.039 INFO:teuthology.nuke:targets: 2021-04-24T00:09:16.040 INFO:teuthology.kill: smithi106.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllKHusPCmgueNxgU2psf2lVm07felMo6pzrJK0ORRefek2hd+GtWk+szBVnERUeaiUKG+Adt0bdCfnO0KeuB37DGmTrC9E7QGndtCRavas8L7tPj1oYFRuXmBqAv/8Z+NBL5ARyofTV8w7CuZiraaWvqeNbpui8j1f2BEigyztomz+J5QJ5MSAFwHa26EG25EorafxjjEr8cnCWA0CsN/gLqj7uQrnUcVcHgmJSZfo2VUIYtus/MJqXJeXxMnXjgqGPia/0zGloPbdiUnDt0t5O3krlFZ9bq3Mf0svNlxmUokZGBywLzpZsvqCHVWpUBHJUb/ADOWt8Sc6/755T6M55hDUDskMtmBGmGob+Xh7KkPUHkuMV9JaF52qh6nNQ3qfjDcbNEsowIPlMewkqyOkBd9tJPqnxHgw+tObArY4Olw8enfuoau2Utk/03gJBPnKxN7uCaE0sRVEzED5Nv/vEd8QHODThhIdIUSf2rhbMyOKW/CwSuiAFUg1rQQ3Tk= 2021-04-24T00:09:16.040 INFO:teuthology.kill: smithi145.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtT/OJXIYBNariV5fKY1PXFx+WXW53P6MDw/c7dKv/U7DZ2TU1B+YE3O8iHbw2gG//9/D3Rlbo9mIIn/8dwvgnXmgh1WGcRl40Ezo0ax+RIEPGQ+DTfTns0M34NtQ5SrpoilJNkog6d82ODjGJRHVVCbGxu2aPx3tQhQByowKFya4LXxN21bNQ0o8KF6ps22joa+Jyrq+qzL9yufeMJ2j1vWLa1IJhdePueUEPnXHdZ2b2QwjksljnvyItabrZg7JbMX9V8/slFTNDo1BkdpVv6XfzdWLE9lWXY1ZePVJxWmSmhhcYH2WlSFnxHbfYd/FaIvv4FcdEaLfO5TrfNr5vqyRWIyttU6OETbUrtzFN0EkuXN0qDSe8Bck9x11liHyttWmEYNkpHZGxV4lXqCVHSYSlufgoGhG0rfKTRXk1LSWx1FC4MGt5Zr+xVOOX8rOmznktWFaaca8v2jL1IyPMroQfbwPZpWBmSVFdUI2k7p9OaoXCEJx4VfnEqtSANLM= 2021-04-24T00:09:16.062 INFO:teuthology.kill:2021-04-24 00:09:16,061.061 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-24T00:09:16.074 INFO:teuthology.kill:2021-04-24 00:09:16,074.074 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-24T00:09:16.143 INFO:teuthology.kill:2021-04-24 00:09:16,142.142 INFO:teuthology.orchestra.console:Power off smithi106 2021-04-24T00:09:16.157 INFO:teuthology.kill:2021-04-24 00:09:16,156.156 INFO:teuthology.orchestra.console:Power off smithi145 2021-04-24T00:09:16.285 INFO:teuthology.kill:2021-04-24 00:09:16,285.285 INFO:teuthology.orchestra.console:Power off for smithi106 completed 2021-04-24T00:09:16.296 INFO:teuthology.kill:2021-04-24 00:09:16,296.296 INFO:teuthology.orchestra.console:Power off for smithi145 completed 2021-04-24T00:09:16.440 INFO:teuthology.kill:2021-04-24 00:09:16,440.440 INFO:teuthology.lock.ops:unlocked smithi106.front.sepia.ceph.com 2021-04-24T00:09:16.459 INFO:teuthology.kill:2021-04-24 00:09:16,459.459 INFO:teuthology.lock.ops:unlocked smithi145.front.sepia.ceph.com 2021-04-24T00:11:53.474 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-04-24T00:11:54.064 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-04-24T00:11:54.064 INFO:teuthology.nuke:Checking targets against current locks 2021-04-24T00:11:54.161 INFO:teuthology.nuke:Not nuking smithi106.front.sepia.ceph.com because description doesn't match 2021-04-24T00:11:54.161 INFO:teuthology.nuke:Not nuking smithi145.front.sepia.ceph.com because description doesn't match