2021-04-23T16:18:18.722 INFO:root:teuthology version: 1.1.0-2713a3cd 2021-04-23T16:18:18.827 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba032.front.sepia.ceph.com' reimaging 2021-04-23T16:18:18.828 INFO:teuthology.lock.ops:Updating [ubuntu@gibba032.front.sepia.ceph.com]: reset os type and version on server 2021-04-23T16:18:18.829 INFO:teuthology.lock.ops:Updating gibba032.front.sepia.ceph.com on lock server 2021-04-23T16:18:18.845 INFO:teuthology.lock.ops:Node 'ubuntu@gibba032.front.sepia.ceph.com' reimaging is complete 2021-04-23T16:18:18.846 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba044.front.sepia.ceph.com' reimaging 2021-04-23T16:18:18.846 INFO:teuthology.lock.ops:Updating [ubuntu@gibba044.front.sepia.ceph.com]: reset os type and version on server 2021-04-23T16:18:18.846 INFO:teuthology.lock.ops:Updating gibba044.front.sepia.ceph.com on lock server 2021-04-23T16:18:18.860 INFO:teuthology.lock.ops:Node 'ubuntu@gibba044.front.sepia.ceph.com' reimaging is complete 2021-04-23T16:18:19.048 INFO:teuthology.provision.fog.gibba032:Scheduling deploy of ubuntu 20.04 2021-04-23T16:18:19.079 INFO:teuthology.provision.fog.gibba044:Scheduling deploy of ubuntu 20.04 2021-04-23T16:18:19.383 INFO:teuthology.orchestra.console:Power off gibba032 2021-04-23T16:18:19.419 INFO:teuthology.orchestra.console:Power off gibba044 2021-04-23T16:18:27.786 INFO:teuthology.orchestra.console:Power off for gibba032 completed 2021-04-23T16:18:27.887 INFO:teuthology.orchestra.console:Power on gibba032 2021-04-23T16:18:31.903 INFO:teuthology.orchestra.console:Power off for gibba044 completed 2021-04-23T16:18:32.004 INFO:teuthology.orchestra.console:Power on gibba044 2021-04-23T16:18:36.265 INFO:teuthology.orchestra.console:Power on for gibba032 completed 2021-04-23T16:18:36.270 INFO:teuthology.orchestra.console:Power on for gibba044 completed 2021-04-23T16:18:36.366 INFO:teuthology.provision.fog.gibba032:Waiting for deploy to finish 2021-04-23T16:18:36.371 INFO:teuthology.provision.fog.gibba044:Waiting for deploy to finish 2021-04-23T16:23:21.832 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-23T16:23:22.295 INFO:teuthology.provision.fog.gibba044:Node is ready 2021-04-23T16:23:22.303 INFO:teuthology.orchestra.run.gibba044.stdout:gibba044.front.sepia.ceph.com 2021-04-23T16:23:22.312 INFO:teuthology.orchestra.run.gibba044.stdout:172.21.2.144 gibba044.front.sepia.ceph.com gibba044 2021-04-23T16:23:22.485 INFO:teuthology.provision.fog.gibba044:Deploy complete! 2021-04-23T16:23:26.019 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-23T16:23:26.457 INFO:teuthology.provision.fog.gibba032:Node is ready 2021-04-23T16:23:26.461 INFO:teuthology.orchestra.run.gibba032.stdout:gibba032.front.sepia.ceph.com 2021-04-23T16:23:26.466 INFO:teuthology.orchestra.run.gibba032.stdout:172.21.2.132 gibba032.front.sepia.ceph.com gibba032 2021-04-23T16:23:26.640 INFO:teuthology.provision.fog.gibba032:Deploy complete! 2021-04-23T16:23:26.771 INFO:teuthology.lock.ops:Checking gibba032.front.sepia.ceph.com 2021-04-23T16:23:26.771 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-23T16:23:26.789 INFO:teuthology.lock.ops:Checking gibba044.front.sepia.ceph.com 2021-04-23T16:23:26.789 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-23T16:23:26.802 INFO:teuthology.lock.ops:Updating [gibba032.front.sepia.ceph.com]: set os type and version on server 2021-04-23T16:23:26.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-23T16:23:26.990 INFO:teuthology.orchestra.run.gibba032.stdout:x86_64 2021-04-23T16:23:26.998 INFO:teuthology.orchestra.run.gibba032.stdout:NAME="Ubuntu" 2021-04-23T16:23:26.998 INFO:teuthology.orchestra.run.gibba032.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-04-23T16:23:26.998 INFO:teuthology.orchestra.run.gibba032.stdout:ID=ubuntu 2021-04-23T16:23:26.998 INFO:teuthology.orchestra.run.gibba032.stdout:ID_LIKE=debian 2021-04-23T16:23:26.998 INFO:teuthology.orchestra.run.gibba032.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-04-23T16:23:26.999 INFO:teuthology.orchestra.run.gibba032.stdout:VERSION_ID="20.04" 2021-04-23T16:23:26.999 INFO:teuthology.orchestra.run.gibba032.stdout:HOME_URL="https://www.ubuntu.com/" 2021-04-23T16:23:26.999 INFO:teuthology.orchestra.run.gibba032.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-04-23T16:23:26.999 INFO:teuthology.orchestra.run.gibba032.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-04-23T16:23:26.999 INFO:teuthology.orchestra.run.gibba032.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-04-23T16:23:26.999 INFO:teuthology.orchestra.run.gibba032.stdout:VERSION_CODENAME=focal 2021-04-23T16:23:26.999 INFO:teuthology.orchestra.run.gibba032.stdout:UBUNTU_CODENAME=focal 2021-04-23T16:23:27.000 INFO:teuthology.lock.ops:Updating gibba032.front.sepia.ceph.com on lock server 2021-04-23T16:23:27.016 INFO:teuthology.lock.ops:Updating [gibba044.front.sepia.ceph.com]: set os type and version on server 2021-04-23T16:23:27.016 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-23T16:23:27.156 INFO:teuthology.orchestra.run.gibba044.stdout:x86_64 2021-04-23T16:23:27.206 INFO:teuthology.orchestra.run.gibba044.stdout:NAME="Ubuntu" 2021-04-23T16:23:27.206 INFO:teuthology.orchestra.run.gibba044.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-04-23T16:23:27.206 INFO:teuthology.orchestra.run.gibba044.stdout:ID=ubuntu 2021-04-23T16:23:27.206 INFO:teuthology.orchestra.run.gibba044.stdout:ID_LIKE=debian 2021-04-23T16:23:27.206 INFO:teuthology.orchestra.run.gibba044.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-04-23T16:23:27.206 INFO:teuthology.orchestra.run.gibba044.stdout:VERSION_ID="20.04" 2021-04-23T16:23:27.206 INFO:teuthology.orchestra.run.gibba044.stdout:HOME_URL="https://www.ubuntu.com/" 2021-04-23T16:23:27.206 INFO:teuthology.orchestra.run.gibba044.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-04-23T16:23:27.207 INFO:teuthology.orchestra.run.gibba044.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-04-23T16:23:27.207 INFO:teuthology.orchestra.run.gibba044.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-04-23T16:23:27.207 INFO:teuthology.orchestra.run.gibba044.stdout:VERSION_CODENAME=focal 2021-04-23T16:23:27.207 INFO:teuthology.orchestra.run.gibba044.stdout:UBUNTU_CODENAME=focal 2021-04-23T16:23:27.207 INFO:teuthology.lock.ops:Updating gibba044.front.sepia.ceph.com on lock server 2021-04-23T16:23:27.281 INFO:teuthology.dispatcher.supervisor:Running job 6067882 2021-04-23T16:23:27.281 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2021-04-23_08:19:18-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6067882 --name rfriedma-2021-04-23_08:19:18-rados-wip-ronenf-scrub-sched-distro-basic-gibba --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-snaps-balanced} -- /home/teuthworker/archive/rfriedma-2021-04-23_08:19:18-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6067882/orig.config.yaml 2021-04-23T16:23:27.285 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2021-04-23_08:19:18-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6067882 2021-04-23T16:23:27.286 INFO:teuthology.dispatcher.supervisor:Job PID: 2391 2021-04-23T16:23:27.286 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-04-24T04:23:45.352 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-04-24T04:23:45.570 INFO:teuthology.kill:Killing Pids: {2391} 2021-04-24T04:23:47.168 INFO:teuthology.kill:Nuking machines: ['gibba032', 'gibba044'] 2021-04-24T04:23:48.102 INFO:teuthology.kill:2021-04-24 04:23:48,101.101 INFO:teuthology.nuke:targets: 2021-04-24T04:23:48.102 INFO:teuthology.kill: gibba032.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvrwtyMaHQoN2xynMNBXHFtPQSliKDvuA93YgWtg4nT7yRZ3eLlD3+ZKdsDOclSVXHO+hYsnXocGlevB/q/8PqyfXsjpi4U8AhLc00bHfs34zgvE2mf3sJjdRyFNumlNIkNyG1szAMfbBlRO1buxIOfovTCrE40OSrFL4r99p/U3UOK1Wa53RG9ex5GrE7TA/ylqTa1RF79DcWrJzc2zhY1Y8PCUny0l7dOMRNb1wt0klY7jL3HCsGlS/WmObYdxHiQlehK7Rr2lVoeFjLYWLMKzML8bMFCjmNYldkde8FBV9mtKPSuLm2HwN9Gp0kkuCOO2YSJ0J6h2SH06McrkTV7DcWRt/rLciPSqkl5dPyKm/q3GOnheMf57RvgSH6LhAwuvQeJo7nJDMhh8wBZIjoT2xMpEtptsdFv9nFuR7s864z8wcftYk7VVvpgwUaEIwOiExLigPuTxAs3fi9wyeB1fynOjHHtFkWShdFPhaafiZDe6FzXr7TLF2LX012iqk= 2021-04-24T04:23:48.102 INFO:teuthology.kill: gibba044.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDL/OUQrYbJV+8ENLANG+Ov3GLhfbwWhlUv5sqpmvRRllaLvCzBbklAI2VWBf4zU9OwMt/oFncMiYoU+ZbjDksXPu52fTaW1C0P1v+r4M8xUFnLttiLMDr6/gSeZkas3nLETfY43WK1n2LBWoGCIvV88AwtJgzrRYKhJG0csJ5ubLfGv2tVAgHiQBnLOfKqCVda/xFc8ZnHHY3QoDPl1xCr3NeEbKbmgRj29cm8UJzF1BsgDXIvE6NvCbVKN2wAiyiZW/M2yYaL2YeBCGO7YSUa7pMGOcHuw/BRqMfZEsKvx7hoYbY2wN+z85+/79pSYCXEnvxdy9xQq7CRoEgSQteBAqSqzU8GCUmy48HpbilLscqrP0xKV0tuzEod9eiePxdseqPhC5b/oGGWxejXF0QkfmjIHGYzGULj9vSrvDS30wm6z810NPnVHTRENSl1aor0BvVcYPSTzYl4MKNSRD5woq0OqS5uIxVLvMA2UkeceKo0sS9t81bKkz98P1jRl+s= 2021-04-24T04:23:48.103 INFO:teuthology.kill:2021-04-24 04:23:48,103.103 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-24T04:23:48.105 INFO:teuthology.kill:2021-04-24 04:23:48,105.105 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-24T04:23:48.210 INFO:teuthology.kill:2021-04-24 04:23:48,209.209 INFO:teuthology.orchestra.console:Power off gibba032 2021-04-24T04:23:48.236 INFO:teuthology.kill:2021-04-24 04:23:48,235.235 INFO:teuthology.orchestra.console:Power off gibba044 2021-04-24T04:23:56.598 INFO:teuthology.kill:2021-04-24 04:23:56,597.597 INFO:teuthology.orchestra.console:Power off for gibba032 completed 2021-04-24T04:23:56.611 INFO:teuthology.kill:2021-04-24 04:23:56,610.610 INFO:teuthology.orchestra.console:Power off for gibba044 completed 2021-04-24T04:23:56.893 INFO:teuthology.task.internal:roles: ubuntu@gibba032.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-04-24T04:23:56.893 INFO:teuthology.task.internal:roles: ubuntu@gibba044.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-04-24T04:23:56.894 INFO:teuthology.misc:Compressing logs... 2021-04-24T04:23:56.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-24T04:24:56.956 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-04-24T04:24:57.022 INFO:teuthology.kill:No teuthology processes running 2021-04-24T04:24:58.206 INFO:teuthology.kill:Nuking machines: ['gibba032', 'gibba044'] 2021-04-24T04:24:59.189 INFO:teuthology.kill:2021-04-24 04:24:59,189.189 INFO:teuthology.nuke:targets: 2021-04-24T04:24:59.189 INFO:teuthology.kill: gibba032.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvrwtyMaHQoN2xynMNBXHFtPQSliKDvuA93YgWtg4nT7yRZ3eLlD3+ZKdsDOclSVXHO+hYsnXocGlevB/q/8PqyfXsjpi4U8AhLc00bHfs34zgvE2mf3sJjdRyFNumlNIkNyG1szAMfbBlRO1buxIOfovTCrE40OSrFL4r99p/U3UOK1Wa53RG9ex5GrE7TA/ylqTa1RF79DcWrJzc2zhY1Y8PCUny0l7dOMRNb1wt0klY7jL3HCsGlS/WmObYdxHiQlehK7Rr2lVoeFjLYWLMKzML8bMFCjmNYldkde8FBV9mtKPSuLm2HwN9Gp0kkuCOO2YSJ0J6h2SH06McrkTV7DcWRt/rLciPSqkl5dPyKm/q3GOnheMf57RvgSH6LhAwuvQeJo7nJDMhh8wBZIjoT2xMpEtptsdFv9nFuR7s864z8wcftYk7VVvpgwUaEIwOiExLigPuTxAs3fi9wyeB1fynOjHHtFkWShdFPhaafiZDe6FzXr7TLF2LX012iqk= 2021-04-24T04:24:59.189 INFO:teuthology.kill: gibba044.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDL/OUQrYbJV+8ENLANG+Ov3GLhfbwWhlUv5sqpmvRRllaLvCzBbklAI2VWBf4zU9OwMt/oFncMiYoU+ZbjDksXPu52fTaW1C0P1v+r4M8xUFnLttiLMDr6/gSeZkas3nLETfY43WK1n2LBWoGCIvV88AwtJgzrRYKhJG0csJ5ubLfGv2tVAgHiQBnLOfKqCVda/xFc8ZnHHY3QoDPl1xCr3NeEbKbmgRj29cm8UJzF1BsgDXIvE6NvCbVKN2wAiyiZW/M2yYaL2YeBCGO7YSUa7pMGOcHuw/BRqMfZEsKvx7hoYbY2wN+z85+/79pSYCXEnvxdy9xQq7CRoEgSQteBAqSqzU8GCUmy48HpbilLscqrP0xKV0tuzEod9eiePxdseqPhC5b/oGGWxejXF0QkfmjIHGYzGULj9vSrvDS30wm6z810NPnVHTRENSl1aor0BvVcYPSTzYl4MKNSRD5woq0OqS5uIxVLvMA2UkeceKo0sS9t81bKkz98P1jRl+s= 2021-04-24T04:25:00.297 INFO:teuthology.kill:2021-04-24 04:25:00,297.297 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-24T04:25:00.309 INFO:teuthology.kill:2021-04-24 04:25:00,309.309 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-24T04:25:00.399 INFO:teuthology.kill:2021-04-24 04:25:00,399.399 INFO:teuthology.orchestra.console:Power off gibba032 2021-04-24T04:25:00.413 INFO:teuthology.kill:2021-04-24 04:25:00,413.413 INFO:teuthology.orchestra.console:Power off gibba044 2021-04-24T04:25:00.549 INFO:teuthology.kill:2021-04-24 04:25:00,549.549 INFO:teuthology.orchestra.console:Power off for gibba032 completed 2021-04-24T04:25:00.562 INFO:teuthology.kill:2021-04-24 04:25:00,562.562 INFO:teuthology.orchestra.console:Power off for gibba044 completed 2021-04-24T04:25:00.702 INFO:teuthology.kill:2021-04-24 04:25:00,701.701 INFO:teuthology.lock.ops:unlocked gibba032.front.sepia.ceph.com 2021-04-24T04:25:00.718 INFO:teuthology.kill:2021-04-24 04:25:00,718.718 INFO:teuthology.lock.ops:unlocked gibba044.front.sepia.ceph.com 2021-04-24T04:27:01.013 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-04-24T04:27:01.125 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-04-24T04:27:01.125 INFO:teuthology.nuke:Checking targets against current locks 2021-04-24T04:27:01.195 INFO:teuthology.nuke:Not nuking gibba044.front.sepia.ceph.com because description doesn't match 2021-04-24T04:27:01.195 INFO:teuthology.nuke:Not nuking gibba032.front.sepia.ceph.com because description doesn't match