2021-04-24T16:49:54.291 INFO:root:teuthology version: 1.1.0-2713a3cd 2021-04-24T16:49:54.381 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba026.front.sepia.ceph.com' reimaging 2021-04-24T16:49:54.382 INFO:teuthology.lock.ops:Updating [ubuntu@gibba026.front.sepia.ceph.com]: reset os type and version on server 2021-04-24T16:49:54.382 INFO:teuthology.lock.ops:Updating gibba026.front.sepia.ceph.com on lock server 2021-04-24T16:49:54.397 INFO:teuthology.lock.ops:Node 'ubuntu@gibba026.front.sepia.ceph.com' reimaging is complete 2021-04-24T16:49:54.397 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba044.front.sepia.ceph.com' reimaging 2021-04-24T16:49:54.397 INFO:teuthology.lock.ops:Updating [ubuntu@gibba044.front.sepia.ceph.com]: reset os type and version on server 2021-04-24T16:49:54.397 INFO:teuthology.lock.ops:Updating gibba044.front.sepia.ceph.com on lock server 2021-04-24T16:49:54.411 INFO:teuthology.lock.ops:Node 'ubuntu@gibba044.front.sepia.ceph.com' reimaging is complete 2021-04-24T16:49:54.594 INFO:teuthology.provision.fog.gibba026:Scheduling deploy of ubuntu 20.04 2021-04-24T16:49:54.614 INFO:teuthology.provision.fog.gibba044:Scheduling deploy of ubuntu 20.04 2021-04-24T16:49:54.908 INFO:teuthology.orchestra.console:Power off gibba026 2021-04-24T16:49:54.921 INFO:teuthology.orchestra.console:Power off gibba044 2021-04-24T16:50:03.298 INFO:teuthology.orchestra.console:Power off for gibba044 completed 2021-04-24T16:50:03.388 INFO:teuthology.orchestra.console:Power off for gibba026 completed 2021-04-24T16:50:03.399 INFO:teuthology.orchestra.console:Power on gibba044 2021-04-24T16:50:03.490 INFO:teuthology.orchestra.console:Power on gibba026 2021-04-24T16:50:11.775 INFO:teuthology.orchestra.console:Power on for gibba044 completed 2021-04-24T16:50:11.864 INFO:teuthology.orchestra.console:Power on for gibba026 completed 2021-04-24T16:50:11.876 INFO:teuthology.provision.fog.gibba044:Waiting for deploy to finish 2021-04-24T16:50:11.966 INFO:teuthology.provision.fog.gibba026:Waiting for deploy to finish 2021-04-24T16:54:51.126 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-24T16:54:51.532 INFO:teuthology.provision.fog.gibba044:Node is ready 2021-04-24T16:54:51.540 INFO:teuthology.orchestra.run.gibba044.stdout:gibba044.front.sepia.ceph.com 2021-04-24T16:54:51.544 INFO:teuthology.orchestra.run.gibba044.stdout:172.21.2.144 gibba044.front.sepia.ceph.com gibba044 2021-04-24T16:54:51.716 INFO:teuthology.provision.fog.gibba044:Deploy complete! 2021-04-24T16:54:54.268 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-24T16:54:54.684 INFO:teuthology.provision.fog.gibba026:Node is ready 2021-04-24T16:54:54.689 INFO:teuthology.orchestra.run.gibba026.stdout:gibba026.front.sepia.ceph.com 2021-04-24T16:54:54.695 INFO:teuthology.orchestra.run.gibba026.stdout:172.21.2.126 gibba026.front.sepia.ceph.com gibba026 2021-04-24T16:54:54.861 INFO:teuthology.provision.fog.gibba026:Deploy complete! 2021-04-24T16:55:16.443 INFO:teuthology.lock.ops:Checking gibba026.front.sepia.ceph.com 2021-04-24T16:55:16.444 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-24T16:55:17.287 INFO:teuthology.lock.ops:Checking gibba044.front.sepia.ceph.com 2021-04-24T16:55:17.287 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-24T16:55:18.023 INFO:teuthology.lock.ops:Updating [gibba026.front.sepia.ceph.com]: set os type and version on server 2021-04-24T16:55:18.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-24T16:55:18.194 INFO:teuthology.orchestra.run.gibba026.stdout:x86_64 2021-04-24T16:55:18.198 INFO:teuthology.orchestra.run.gibba026.stdout:NAME="Ubuntu" 2021-04-24T16:55:18.198 INFO:teuthology.orchestra.run.gibba026.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-04-24T16:55:18.198 INFO:teuthology.orchestra.run.gibba026.stdout:ID=ubuntu 2021-04-24T16:55:18.198 INFO:teuthology.orchestra.run.gibba026.stdout:ID_LIKE=debian 2021-04-24T16:55:18.198 INFO:teuthology.orchestra.run.gibba026.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-04-24T16:55:18.198 INFO:teuthology.orchestra.run.gibba026.stdout:VERSION_ID="20.04" 2021-04-24T16:55:18.198 INFO:teuthology.orchestra.run.gibba026.stdout:HOME_URL="https://www.ubuntu.com/" 2021-04-24T16:55:18.198 INFO:teuthology.orchestra.run.gibba026.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-04-24T16:55:18.198 INFO:teuthology.orchestra.run.gibba026.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-04-24T16:55:18.199 INFO:teuthology.orchestra.run.gibba026.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-04-24T16:55:18.199 INFO:teuthology.orchestra.run.gibba026.stdout:VERSION_CODENAME=focal 2021-04-24T16:55:18.199 INFO:teuthology.orchestra.run.gibba026.stdout:UBUNTU_CODENAME=focal 2021-04-24T16:55:18.200 INFO:teuthology.lock.ops:Updating gibba026.front.sepia.ceph.com on lock server 2021-04-24T16:55:18.225 INFO:teuthology.lock.ops:Updating [gibba044.front.sepia.ceph.com]: set os type and version on server 2021-04-24T16:55:18.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-24T16:55:18.387 INFO:teuthology.orchestra.run.gibba044.stdout:x86_64 2021-04-24T16:55:18.395 INFO:teuthology.orchestra.run.gibba044.stdout:NAME="Ubuntu" 2021-04-24T16:55:18.395 INFO:teuthology.orchestra.run.gibba044.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-04-24T16:55:18.395 INFO:teuthology.orchestra.run.gibba044.stdout:ID=ubuntu 2021-04-24T16:55:18.395 INFO:teuthology.orchestra.run.gibba044.stdout:ID_LIKE=debian 2021-04-24T16:55:18.395 INFO:teuthology.orchestra.run.gibba044.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-04-24T16:55:18.396 INFO:teuthology.orchestra.run.gibba044.stdout:VERSION_ID="20.04" 2021-04-24T16:55:18.396 INFO:teuthology.orchestra.run.gibba044.stdout:HOME_URL="https://www.ubuntu.com/" 2021-04-24T16:55:18.396 INFO:teuthology.orchestra.run.gibba044.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-04-24T16:55:18.396 INFO:teuthology.orchestra.run.gibba044.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-04-24T16:55:18.396 INFO:teuthology.orchestra.run.gibba044.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-04-24T16:55:18.396 INFO:teuthology.orchestra.run.gibba044.stdout:VERSION_CODENAME=focal 2021-04-24T16:55:18.396 INFO:teuthology.orchestra.run.gibba044.stdout:UBUNTU_CODENAME=focal 2021-04-24T16:55:18.396 INFO:teuthology.lock.ops:Updating gibba044.front.sepia.ceph.com on lock server 2021-04-24T16:55:18.478 INFO:teuthology.dispatcher.supervisor:Running job 6069513 2021-04-24T16:55:18.478 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2021-04-23_22:57:33-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6069513 --name rfriedma-2021-04-23_22:57:33-rados-wip-ronenf-scrub-sched-distro-basic-gibba --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-pool-snaps} -- /home/teuthworker/archive/rfriedma-2021-04-23_22:57:33-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6069513/orig.config.yaml 2021-04-24T16:55:18.483 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2021-04-23_22:57:33-rados-wip-ronenf-scrub-sched-distro-basic-gibba/6069513 2021-04-24T16:55:18.483 INFO:teuthology.dispatcher.supervisor:Job PID: 23949 2021-04-24T16:55:18.483 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-04-25T04:56:56.539 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-04-25T04:56:56.756 INFO:teuthology.kill:Killing Pids: {23949} 2021-04-25T04:56:57.797 INFO:teuthology.kill:Nuking machines: ['gibba026', 'gibba044'] 2021-04-25T04:56:58.701 INFO:teuthology.kill:2021-04-25 04:56:58,701.701 INFO:teuthology.nuke:targets: 2021-04-25T04:56:58.702 INFO:teuthology.kill: gibba026.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrJEF9LJ1e6MMQuYjQ6cZCCDNISnDrx5XOJr0Lpp6X8MYyss+qhxbLZMycMGpLvtikVXKAnv1bN8HIbcmy8bSWIZ1og7hHEB8ZLhzV5OBvYl7Uags08U4CDb1G2/hpUiFt17zMT+PBNP4QHfl0weSNJozkNxetvHEZLtaws3o4oPBqbW6hmRu2IaEB5TlJiV1OY7MK86JfmMlLvDXprZKPn2Kpdo0Wdx4NJ0L+ejKfGbWc0eseR5vjt/uiy5hl4dXcK0cUg0CWaCaiOQ6cYcdy+/5wH1nMuyXUDu5PXo+UYPdMH8CAwzb/OjDgneMMa8JV3Wnei501pdrpKQCYrV2nHKyvzs3zM5IbrZYU6J7TavM51zMBa0tGqaxZxsTtQE1OZhlAbaK2GRgJO5ge82ADIhtyPsQRnQZhzeRg9FLI7KGddtwmzor3IviUnXVLbcuKeljBD73dAEtDK2S4AnVXOr4OFHJq+Y3nxUp5bICvIBXyg6VqlkirLopwwnuJLxM= 2021-04-25T04:56:58.702 INFO:teuthology.kill: gibba044.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwfohKOkzdQufGpen9coaC2YR3WL8jhMqy+x2BlfyTAz310YssPUzkW8qT6eyutH8Lz8GJhCTqM8T0bWSRrDfNqUUQKZ/F2fcVIaugzsm70D9pR3uROOpIPHUnfmSPwhzfd66K5EyYQjmc7K4o32/wZFxRzjglSuKg8XTU7m8wxajF4OQlL0BISmQDNcJi92X2aCZ6VNAjamuILMRy9AEwD2YSX00PP8RrwwPm0fIZmPQoQA7jMRyB5ywZHVunnK+Hf1cjoGzV9UsqUAV5qw3StBZead5XdJQ07rgi8rBHAze1wzWyGBRPuWyxWS1ZGoNI9oqxyJrmfVr9fQkeq61MFsaFvyf+p4onmLI5NvBf1aM+W3hN/e6gXXKeFN2RYqP4i5xBFbIvdZ0kLbWmMwP12SgvjqzjkxkudTjIAewcIwRXnGn3btFunVeoTB7S5zsdqIgpVqVvwSZBuAmfpFKaXc0MMTYJzL8w+eZ3bQB1EBRNnsWqTKypwCpnjBRb37U= 2021-04-25T04:56:58.702 INFO:teuthology.kill:2021-04-25 04:56:58,701.701 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-25T04:56:58.704 INFO:teuthology.kill:2021-04-25 04:56:58,703.703 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-25T04:56:58.777 INFO:teuthology.kill:2021-04-25 04:56:58,777.777 INFO:teuthology.orchestra.console:Power off gibba026 2021-04-25T04:56:58.789 INFO:teuthology.kill:2021-04-25 04:56:58,789.789 INFO:teuthology.orchestra.console:Power off gibba044 2021-04-25T04:57:07.155 INFO:teuthology.kill:2021-04-25 04:57:07,155.155 INFO:teuthology.orchestra.console:Power off for gibba026 completed 2021-04-25T04:57:11.272 INFO:teuthology.kill:2021-04-25 04:57:11,272.272 INFO:teuthology.orchestra.console:Power off for gibba044 completed 2021-04-25T04:57:11.484 INFO:teuthology.task.internal:roles: ubuntu@gibba026.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-04-25T04:57:11.484 INFO:teuthology.task.internal:roles: ubuntu@gibba044.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-04-25T04:57:11.484 INFO:teuthology.misc:Compressing logs... 2021-04-25T04:57:11.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-25T04:57:46.076 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-04-25T04:57:46.141 INFO:teuthology.kill:No teuthology processes running 2021-04-25T04:57:47.145 INFO:teuthology.kill:Nuking machines: ['gibba026', 'gibba044'] 2021-04-25T04:57:48.018 INFO:teuthology.kill:2021-04-25 04:57:48,017.017 INFO:teuthology.nuke:targets: 2021-04-25T04:57:48.018 INFO:teuthology.kill: gibba026.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrJEF9LJ1e6MMQuYjQ6cZCCDNISnDrx5XOJr0Lpp6X8MYyss+qhxbLZMycMGpLvtikVXKAnv1bN8HIbcmy8bSWIZ1og7hHEB8ZLhzV5OBvYl7Uags08U4CDb1G2/hpUiFt17zMT+PBNP4QHfl0weSNJozkNxetvHEZLtaws3o4oPBqbW6hmRu2IaEB5TlJiV1OY7MK86JfmMlLvDXprZKPn2Kpdo0Wdx4NJ0L+ejKfGbWc0eseR5vjt/uiy5hl4dXcK0cUg0CWaCaiOQ6cYcdy+/5wH1nMuyXUDu5PXo+UYPdMH8CAwzb/OjDgneMMa8JV3Wnei501pdrpKQCYrV2nHKyvzs3zM5IbrZYU6J7TavM51zMBa0tGqaxZxsTtQE1OZhlAbaK2GRgJO5ge82ADIhtyPsQRnQZhzeRg9FLI7KGddtwmzor3IviUnXVLbcuKeljBD73dAEtDK2S4AnVXOr4OFHJq+Y3nxUp5bICvIBXyg6VqlkirLopwwnuJLxM= 2021-04-25T04:57:48.018 INFO:teuthology.kill: gibba044.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwfohKOkzdQufGpen9coaC2YR3WL8jhMqy+x2BlfyTAz310YssPUzkW8qT6eyutH8Lz8GJhCTqM8T0bWSRrDfNqUUQKZ/F2fcVIaugzsm70D9pR3uROOpIPHUnfmSPwhzfd66K5EyYQjmc7K4o32/wZFxRzjglSuKg8XTU7m8wxajF4OQlL0BISmQDNcJi92X2aCZ6VNAjamuILMRy9AEwD2YSX00PP8RrwwPm0fIZmPQoQA7jMRyB5ywZHVunnK+Hf1cjoGzV9UsqUAV5qw3StBZead5XdJQ07rgi8rBHAze1wzWyGBRPuWyxWS1ZGoNI9oqxyJrmfVr9fQkeq61MFsaFvyf+p4onmLI5NvBf1aM+W3hN/e6gXXKeFN2RYqP4i5xBFbIvdZ0kLbWmMwP12SgvjqzjkxkudTjIAewcIwRXnGn3btFunVeoTB7S5zsdqIgpVqVvwSZBuAmfpFKaXc0MMTYJzL8w+eZ3bQB1EBRNnsWqTKypwCpnjBRb37U= 2021-04-25T04:57:48.038 INFO:teuthology.kill:2021-04-25 04:57:48,038.038 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-25T04:57:48.065 INFO:teuthology.kill:2021-04-25 04:57:48,065.065 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-25T04:57:48.131 INFO:teuthology.kill:2021-04-25 04:57:48,131.131 INFO:teuthology.orchestra.console:Power off gibba026 2021-04-25T04:57:48.142 INFO:teuthology.kill:2021-04-25 04:57:48,142.142 INFO:teuthology.orchestra.console:Power off gibba044 2021-04-25T04:57:48.276 INFO:teuthology.kill:2021-04-25 04:57:48,275.275 INFO:teuthology.orchestra.console:Power off for gibba026 completed 2021-04-25T04:57:48.288 INFO:teuthology.kill:2021-04-25 04:57:48,288.288 INFO:teuthology.orchestra.console:Power off for gibba044 completed 2021-04-25T04:57:48.423 INFO:teuthology.kill:2021-04-25 04:57:48,422.422 INFO:teuthology.lock.ops:unlocked gibba026.front.sepia.ceph.com 2021-04-25T04:57:48.439 INFO:teuthology.kill:2021-04-25 04:57:48,439.439 INFO:teuthology.lock.ops:unlocked gibba044.front.sepia.ceph.com 2021-04-25T05:00:09.481 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-04-25T05:00:09.687 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-04-25T05:00:09.687 INFO:teuthology.nuke:Checking targets against current locks 2021-04-25T05:00:09.808 INFO:teuthology.nuke:Not nuking gibba026.front.sepia.ceph.com because description doesn't match 2021-04-25T05:00:09.808 INFO:teuthology.nuke:Not nuking gibba044.front.sepia.ceph.com because description doesn't match