2021-09-14T06:44:08.858 INFO:root:teuthology version: 1.1.0-b135a4f8 2021-09-14T06:44:08.983 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi084.front.sepia.ceph.com' reimaging 2021-09-14T06:44:08.984 INFO:teuthology.lock.ops:Updating [ubuntu@smithi084.front.sepia.ceph.com]: reset os type and version on server 2021-09-14T06:44:08.985 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2021-09-14T06:44:09.003 INFO:teuthology.lock.ops:Node 'ubuntu@smithi084.front.sepia.ceph.com' reimaging is complete 2021-09-14T06:44:09.003 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi142.front.sepia.ceph.com' reimaging 2021-09-14T06:44:09.004 INFO:teuthology.lock.ops:Updating [ubuntu@smithi142.front.sepia.ceph.com]: reset os type and version on server 2021-09-14T06:44:09.004 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2021-09-14T06:44:09.029 INFO:teuthology.lock.ops:Node 'ubuntu@smithi142.front.sepia.ceph.com' reimaging is complete 2021-09-14T06:44:09.235 INFO:teuthology.provision.fog.smithi084:Scheduling deploy of ubuntu 20.04 2021-09-14T06:44:09.240 INFO:teuthology.provision.fog.smithi142:Scheduling deploy of ubuntu 20.04 2021-09-14T06:44:09.611 INFO:teuthology.orchestra.console:Power off smithi084 2021-09-14T06:44:09.630 INFO:teuthology.orchestra.console:Power off smithi142 2021-09-14T06:44:22.102 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2021-09-14T06:44:22.109 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2021-09-14T06:44:22.204 INFO:teuthology.orchestra.console:Power on smithi084 2021-09-14T06:44:22.213 INFO:teuthology.orchestra.console:Power on smithi142 2021-09-14T06:44:26.464 INFO:teuthology.orchestra.console:Power on for smithi084 completed 2021-09-14T06:44:26.565 INFO:teuthology.provision.fog.smithi084:Waiting for deploy to finish 2021-09-14T06:44:30.587 INFO:teuthology.orchestra.console:Power on for smithi142 completed 2021-09-14T06:44:30.688 INFO:teuthology.provision.fog.smithi142:Waiting for deploy to finish 2021-09-14T06:49:24.480 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-14T06:49:24.932 INFO:teuthology.provision.fog.smithi084:Node is ready 2021-09-14T06:49:24.939 INFO:teuthology.orchestra.run.smithi084.stdout:smithi084.front.sepia.ceph.com 2021-09-14T06:49:24.946 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84 smithi084.front.sepia.ceph.com smithi084 2021-09-14T06:49:25.118 INFO:teuthology.provision.fog.smithi084:Deploy complete! 2021-09-14T06:49:25.495 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-14T06:49:25.934 INFO:teuthology.provision.fog.smithi142:Node is ready 2021-09-14T06:49:25.991 INFO:teuthology.orchestra.run.smithi142.stdout:smithi142.front.sepia.ceph.com 2021-09-14T06:49:26.045 INFO:teuthology.orchestra.run.smithi142.stdout:172.21.15.142 smithi142.front.sepia.ceph.com smithi142 2021-09-14T06:49:26.185 INFO:teuthology.provision.fog.smithi142:Deploy complete! 2021-09-14T06:49:26.356 INFO:teuthology.lock.ops:Checking smithi084.front.sepia.ceph.com 2021-09-14T06:49:26.356 INFO:teuthology.lock.ops:New key found. Updating... 2021-09-14T06:49:26.376 INFO:teuthology.lock.ops:Checking smithi142.front.sepia.ceph.com 2021-09-14T06:49:26.376 INFO:teuthology.lock.ops:New key found. Updating... 2021-09-14T06:49:26.391 INFO:teuthology.lock.ops:Updating [smithi084.front.sepia.ceph.com]: set os type and version on server 2021-09-14T06:49:26.392 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-14T06:49:26.557 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2021-09-14T06:49:26.607 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="Ubuntu" 2021-09-14T06:49:26.607 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-09-14T06:49:26.608 INFO:teuthology.orchestra.run.smithi084.stdout:ID=ubuntu 2021-09-14T06:49:26.608 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE=debian 2021-09-14T06:49:26.608 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-09-14T06:49:26.608 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="20.04" 2021-09-14T06:49:26.609 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://www.ubuntu.com/" 2021-09-14T06:49:26.609 INFO:teuthology.orchestra.run.smithi084.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-09-14T06:49:26.609 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-09-14T06:49:26.609 INFO:teuthology.orchestra.run.smithi084.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-09-14T06:49:26.609 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_CODENAME=focal 2021-09-14T06:49:26.610 INFO:teuthology.orchestra.run.smithi084.stdout:UBUNTU_CODENAME=focal 2021-09-14T06:49:26.611 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2021-09-14T06:49:26.626 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: set os type and version on server 2021-09-14T06:49:26.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-14T06:49:26.790 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2021-09-14T06:49:26.796 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="Ubuntu" 2021-09-14T06:49:26.796 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-09-14T06:49:26.796 INFO:teuthology.orchestra.run.smithi142.stdout:ID=ubuntu 2021-09-14T06:49:26.797 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE=debian 2021-09-14T06:49:26.797 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-09-14T06:49:26.797 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="20.04" 2021-09-14T06:49:26.797 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://www.ubuntu.com/" 2021-09-14T06:49:26.798 INFO:teuthology.orchestra.run.smithi142.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-09-14T06:49:26.798 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-09-14T06:49:26.798 INFO:teuthology.orchestra.run.smithi142.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-09-14T06:49:26.798 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_CODENAME=focal 2021-09-14T06:49:26.798 INFO:teuthology.orchestra.run.smithi142.stdout:UBUNTU_CODENAME=focal 2021-09-14T06:49:26.799 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2021-09-14T06:49:26.875 INFO:teuthology.dispatcher.supervisor:Running job 6389479 2021-09-14T06:49:26.875 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2021-09-14_06:20:00-rados-wip-ronenf-unique-scrub-distro-basic-smithi/6389479 --name rfriedma-2021-09-14_06:20:00-rados-wip-ronenf-unique-scrub-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-localized} -- /home/teuthworker/archive/rfriedma-2021-09-14_06:20:00-rados-wip-ronenf-unique-scrub-distro-basic-smithi/6389479/orig.config.yaml 2021-09-14T06:49:26.880 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2021-09-14_06:20:00-rados-wip-ronenf-unique-scrub-distro-basic-smithi/6389479 2021-09-14T06:49:26.881 INFO:teuthology.dispatcher.supervisor:Job PID: 25622 2021-09-14T06:49:26.881 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-09-14T18:50:17.520 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-09-14T18:50:18.416 INFO:teuthology.kill:Killing Pids: {25622} 2021-09-14T18:50:19.712 INFO:teuthology.kill:Nuking machines: ['smithi084', 'smithi142'] 2021-09-14T18:50:20.656 INFO:teuthology.kill:2021-09-14 18:50:20,655.655 INFO:teuthology.nuke:targets: 2021-09-14T18:50:20.657 INFO:teuthology.kill: smithi084.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlKQKe2A2kHYj6xf41sWbEr22w4Ab3v/TOhpH/SFeBlM2MWfinqNHiRDZ34dm0+Qo/YSNKK3n6BP3ssuRLrWMNm7oZq0dM4DH/S1FxFSzIuoMXxMVNcReU4ezrQXBgK2kW2oakGng0iVtv1rL07xKSZOVaSr72FIGlp3GtDKe9p2uq4K8Dv0bc9zSIeStDtkZs9ytK4PgZd0pVPIKu0Lx8MX5EFOTjk0sSMFYZbSYJy/2PrZRqZA8lb0JRbmq7hKBv5M+BT4SNsh2LwWXM2LhV+QAtHOXqk3L6Bq8h8w5P20+65EXkDvyEvBrLSnSEHrxOgGyBK9M4uU7v07MmvyVMzTUWYkSRJDAANvO7+gHymJcDZyxovOBtUYvbBHegzG5VP5A5XVwCTteVw2Mo7OtGmx1A2bPLDH7A2NgP8rwI3NNo1J8sV7NU8YLYovDv8nkZpjtduBqudsVRAcMiWncDIE5Z4UA4k8Jpq9C0Y3G2kzHwdDkAKY4Gt89kehQM/GM= 2021-09-14T18:50:20.658 INFO:teuthology.kill: smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDx46Epm7zLbM/SlXkRYtz2/ZmcEAdYdGck/7U/xJxT7XWWPZ9wHlYIFQM11jRpkjojmpBSdCDxudWuaUzGWRdWRJz6R5VL6AyvMut9D7ZyZ/ezEHjSdvI+ZagdhbMSmzSNtZriiulNpMDP1yVBwiK8y3GEGHNmWqS4ebwgFZjz/i72dHrF6ID+DPV9GTXtQ0FUOnw2JhbU8lxRGG5Y5tzLlLWAV/lriUvHl2AG6T/LPFsbYK+310K7RkXAeZpOia+FG4WzmWSJ9bH3z0m+jZGEEAEWoVaJl+vS41Ucubl70BdJLqFEiwFn7ylTb56Gi3A9Udk7dlnl+y1iLe3FjfOBYlJYIVnqaMf5p7b2ys/pUI8ygpk0qwhz5qU1Gz6D3LG4Zzz9RMcB5l1/DfEPoejmyIBvawuBBoonNKgbxqzvptE/uqnPTrAELQ86qqA2c4pc6R/j34S0GhKryYo1krmO1f1MljOHrEbzl5di/W+av9h1wgGLSDUyc+E34TMDAFM= 2021-09-14T18:50:20.658 INFO:teuthology.kill:2021-09-14 18:50:20,656.656 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-14T18:50:20.659 INFO:teuthology.kill:2021-09-14 18:50:20,658.658 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-14T18:50:20.720 INFO:teuthology.kill:2021-09-14 18:50:20,720.720 INFO:teuthology.orchestra.console:Power off smithi084 2021-09-14T18:50:20.734 INFO:teuthology.kill:2021-09-14 18:50:20,734.734 INFO:teuthology.orchestra.console:Power off smithi142 2021-09-14T18:50:29.096 INFO:teuthology.kill:2021-09-14 18:50:29,095.095 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2021-09-14T18:50:33.200 INFO:teuthology.kill:2021-09-14 18:50:33,200.200 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2021-09-14T18:50:33.446 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-09-14T18:50:33.447 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-09-14T18:50:33.453 INFO:teuthology.misc:Compressing logs... 2021-09-14T18:50:33.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-14T18:51:08.116 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-09-14T18:51:08.190 INFO:teuthology.kill:No teuthology processes running 2021-09-14T18:51:09.266 INFO:teuthology.kill:Nuking machines: ['smithi084', 'smithi142'] 2021-09-14T18:51:10.180 INFO:teuthology.kill:2021-09-14 18:51:10,180.180 INFO:teuthology.nuke:targets: 2021-09-14T18:51:10.181 INFO:teuthology.kill: smithi084.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlKQKe2A2kHYj6xf41sWbEr22w4Ab3v/TOhpH/SFeBlM2MWfinqNHiRDZ34dm0+Qo/YSNKK3n6BP3ssuRLrWMNm7oZq0dM4DH/S1FxFSzIuoMXxMVNcReU4ezrQXBgK2kW2oakGng0iVtv1rL07xKSZOVaSr72FIGlp3GtDKe9p2uq4K8Dv0bc9zSIeStDtkZs9ytK4PgZd0pVPIKu0Lx8MX5EFOTjk0sSMFYZbSYJy/2PrZRqZA8lb0JRbmq7hKBv5M+BT4SNsh2LwWXM2LhV+QAtHOXqk3L6Bq8h8w5P20+65EXkDvyEvBrLSnSEHrxOgGyBK9M4uU7v07MmvyVMzTUWYkSRJDAANvO7+gHymJcDZyxovOBtUYvbBHegzG5VP5A5XVwCTteVw2Mo7OtGmx1A2bPLDH7A2NgP8rwI3NNo1J8sV7NU8YLYovDv8nkZpjtduBqudsVRAcMiWncDIE5Z4UA4k8Jpq9C0Y3G2kzHwdDkAKY4Gt89kehQM/GM= 2021-09-14T18:51:10.183 INFO:teuthology.kill: smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDx46Epm7zLbM/SlXkRYtz2/ZmcEAdYdGck/7U/xJxT7XWWPZ9wHlYIFQM11jRpkjojmpBSdCDxudWuaUzGWRdWRJz6R5VL6AyvMut9D7ZyZ/ezEHjSdvI+ZagdhbMSmzSNtZriiulNpMDP1yVBwiK8y3GEGHNmWqS4ebwgFZjz/i72dHrF6ID+DPV9GTXtQ0FUOnw2JhbU8lxRGG5Y5tzLlLWAV/lriUvHl2AG6T/LPFsbYK+310K7RkXAeZpOia+FG4WzmWSJ9bH3z0m+jZGEEAEWoVaJl+vS41Ucubl70BdJLqFEiwFn7ylTb56Gi3A9Udk7dlnl+y1iLe3FjfOBYlJYIVnqaMf5p7b2ys/pUI8ygpk0qwhz5qU1Gz6D3LG4Zzz9RMcB5l1/DfEPoejmyIBvawuBBoonNKgbxqzvptE/uqnPTrAELQ86qqA2c4pc6R/j34S0GhKryYo1krmO1f1MljOHrEbzl5di/W+av9h1wgGLSDUyc+E34TMDAFM= 2021-09-14T18:51:10.202 INFO:teuthology.kill:2021-09-14 18:51:10,202.202 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-14T18:51:10.205 INFO:teuthology.kill:2021-09-14 18:51:10,205.205 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-14T18:51:10.264 INFO:teuthology.kill:2021-09-14 18:51:10,264.264 INFO:teuthology.orchestra.console:Power off smithi142 2021-09-14T18:51:10.277 INFO:teuthology.kill:2021-09-14 18:51:10,277.277 INFO:teuthology.orchestra.console:Power off smithi084 2021-09-14T18:51:10.407 INFO:teuthology.kill:2021-09-14 18:51:10,406.406 INFO:teuthology.orchestra.console:Power off for smithi142 completed 2021-09-14T18:51:10.418 INFO:teuthology.kill:2021-09-14 18:51:10,418.418 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2021-09-14T18:51:10.549 INFO:teuthology.kill:2021-09-14 18:51:10,549.549 INFO:teuthology.lock.ops:unlocked smithi142.front.sepia.ceph.com 2021-09-14T18:51:10.557 INFO:teuthology.kill:2021-09-14 18:51:10,557.557 INFO:teuthology.lock.ops:unlocked smithi084.front.sepia.ceph.com 2021-09-14T18:53:10.856 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-09-14T18:53:10.973 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-09-14T18:53:10.975 INFO:teuthology.nuke:Checking targets against current locks 2021-09-14T18:53:11.163 INFO:teuthology.nuke:Not nuking smithi084.front.sepia.ceph.com because description doesn't match 2021-09-14T18:53:11.164 INFO:teuthology.nuke:Not nuking smithi142.front.sepia.ceph.com because description doesn't match