2021-07-27T21:25:55.292 INFO:root:teuthology version: 1.1.0-73aa7e3b
2021-07-27T21:25:55.397 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi031.front.sepia.ceph.com' reimaging
2021-07-27T21:25:55.398 INFO:teuthology.lock.ops:Updating [ubuntu@smithi031.front.sepia.ceph.com]: reset os type and version on server
2021-07-27T21:25:55.399 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server
2021-07-27T21:25:55.414 INFO:teuthology.lock.ops:Node 'ubuntu@smithi031.front.sepia.ceph.com' reimaging is complete
2021-07-27T21:25:55.415 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi164.front.sepia.ceph.com' reimaging
2021-07-27T21:25:55.416 INFO:teuthology.lock.ops:Updating [ubuntu@smithi164.front.sepia.ceph.com]: reset os type and version on server
2021-07-27T21:25:55.416 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server
2021-07-27T21:25:55.432 INFO:teuthology.lock.ops:Node 'ubuntu@smithi164.front.sepia.ceph.com' reimaging is complete
2021-07-27T21:25:55.619 INFO:teuthology.provision.fog.smithi031:Scheduling deploy of ubuntu 20.04
2021-07-27T21:25:55.642 INFO:teuthology.provision.fog.smithi164:Scheduling deploy of ubuntu 20.04
2021-07-27T21:25:55.985 INFO:teuthology.orchestra.console:Power off smithi031
2021-07-27T21:25:56.000 INFO:teuthology.orchestra.console:Power off smithi164
2021-07-27T21:26:04.358 INFO:teuthology.orchestra.console:Power off for smithi031 completed
2021-07-27T21:26:04.460 INFO:teuthology.orchestra.console:Power on smithi031
2021-07-27T21:26:08.478 INFO:teuthology.orchestra.console:Power off for smithi164 completed
2021-07-27T21:26:08.581 INFO:teuthology.orchestra.console:Power on smithi164
2021-07-27T21:26:12.836 INFO:teuthology.orchestra.console:Power on for smithi164 completed
2021-07-27T21:26:12.853 INFO:teuthology.orchestra.console:Power on for smithi031 completed
2021-07-27T21:26:12.938 INFO:teuthology.provision.fog.smithi164:Waiting for deploy to finish
2021-07-27T21:26:13.010 INFO:teuthology.provision.fog.smithi031:Waiting for deploy to finish
2021-07-27T21:30:51.248 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-07-27T21:30:51.740 INFO:teuthology.provision.fog.smithi031:Node is ready
2021-07-27T21:30:51.747 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031.front.sepia.ceph.com
2021-07-27T21:30:51.755 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31 smithi031.front.sepia.ceph.com smithi031
2021-07-27T21:30:51.984 INFO:teuthology.provision.fog.smithi031:Deploy complete!
2021-07-27T21:31:22.888 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-07-27T21:31:23.327 INFO:teuthology.provision.fog.smithi164:Node is ready
2021-07-27T21:31:23.334 INFO:teuthology.orchestra.run.smithi164.stdout:smithi164.front.sepia.ceph.com
2021-07-27T21:31:23.385 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.15.164 smithi164.front.sepia.ceph.com smithi164
2021-07-27T21:31:23.586 INFO:teuthology.provision.fog.smithi164:Deploy complete!
2021-07-27T21:31:23.758 INFO:teuthology.lock.ops:Checking smithi031.front.sepia.ceph.com
2021-07-27T21:31:23.759 INFO:teuthology.lock.ops:New key found. Updating...
2021-07-27T21:31:23.776 INFO:teuthology.lock.ops:Checking smithi164.front.sepia.ceph.com
2021-07-27T21:31:23.776 INFO:teuthology.lock.ops:New key found. Updating...
2021-07-27T21:31:23.791 INFO:teuthology.lock.ops:Updating [smithi031.front.sepia.ceph.com]: set os type and version on server
2021-07-27T21:31:23.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-07-27T21:31:24.107 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64
2021-07-27T21:31:24.114 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="Ubuntu"
2021-07-27T21:31:24.114 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="20.04.2 LTS (Focal Fossa)"
2021-07-27T21:31:24.114 INFO:teuthology.orchestra.run.smithi031.stdout:ID=ubuntu
2021-07-27T21:31:24.115 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE=debian
2021-07-27T21:31:24.115 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS"
2021-07-27T21:31:24.115 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="20.04"
2021-07-27T21:31:24.115 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://www.ubuntu.com/"
2021-07-27T21:31:24.116 INFO:teuthology.orchestra.run.smithi031.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-07-27T21:31:24.116 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-07-27T21:31:24.116 INFO:teuthology.orchestra.run.smithi031.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-07-27T21:31:24.116 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_CODENAME=focal
2021-07-27T21:31:24.117 INFO:teuthology.orchestra.run.smithi031.stdout:UBUNTU_CODENAME=focal
2021-07-27T21:31:24.118 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server
2021-07-27T21:31:24.133 INFO:teuthology.lock.ops:Updating [smithi164.front.sepia.ceph.com]: set os type and version on server
2021-07-27T21:31:24.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-07-27T21:31:24.306 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64
2021-07-27T21:31:24.313 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="Ubuntu"
2021-07-27T21:31:24.315 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="20.04.2 LTS (Focal Fossa)"
2021-07-27T21:31:24.315 INFO:teuthology.orchestra.run.smithi164.stdout:ID=ubuntu
2021-07-27T21:31:24.315 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE=debian
2021-07-27T21:31:24.316 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS"
2021-07-27T21:31:24.316 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="20.04"
2021-07-27T21:31:24.316 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://www.ubuntu.com/"
2021-07-27T21:31:24.316 INFO:teuthology.orchestra.run.smithi164.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-07-27T21:31:24.317 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-07-27T21:31:24.317 INFO:teuthology.orchestra.run.smithi164.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-07-27T21:31:24.317 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_CODENAME=focal
2021-07-27T21:31:24.317 INFO:teuthology.orchestra.run.smithi164.stdout:UBUNTU_CODENAME=focal
2021-07-27T21:31:24.318 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server
2021-07-27T21:31:24.391 INFO:teuthology.dispatcher.supervisor:Running job 6296341
2021-07-27T21:31:24.392 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2021-07-27_13:23:42-rados-wip_gbenhano_ncb_fix2-distro-basic-smithi/6296341 --name benhanokh-2021-07-27_13:23:42-rados-wip_gbenhano_ncb_fix2-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} -- /home/teuthworker/archive/benhanokh-2021-07-27_13:23:42-rados-wip_gbenhano_ncb_fix2-distro-basic-smithi/6296341/orig.config.yaml
2021-07-27T21:31:24.396 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2021-07-27_13:23:42-rados-wip_gbenhano_ncb_fix2-distro-basic-smithi/6296341
2021-07-27T21:31:24.397 INFO:teuthology.dispatcher.supervisor:Job PID: 32446
2021-07-27T21:31:24.398 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-07-28T09:32:13.714 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-07-28T09:32:13.920 INFO:teuthology.kill:Killing Pids: {32446}
2021-07-28T09:32:14.976 INFO:teuthology.kill:Nuking machines: ['smithi031', 'smithi164']
2021-07-28T09:32:15.929 INFO:teuthology.kill:2021-07-28 09:32:15,929.929 INFO:teuthology.nuke:targets:
2021-07-28T09:32:15.930 INFO:teuthology.kill:  smithi031.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbPKoeaQzW9qqtQ2uLy1RT0aGI8pmfrPxat/BcPyb7vg+Ahglizbkv5FO8G9H7sS1IgWyxKUA48Qdlp4pZzlHQ6aAMLQX1FFao5vpRr1m9u5BPQ4Dk/cF1AyWLV8C6aZozAwdZ5MMpMewGQtN4JYJ8673Bn+DdVBmYIapFl9WSRIcxtjjCfi1YDZQmJ0azL55ymgLFUcIIom1obF9j2Y542UITWodLIBa6m/9XgnCBRf6Y+BBaKIo4Xt1a6/7JBrk5NWengWOl8EXRNU5lnJUr0qvl5doG3fJs/ECS67DWg63XgMKmvHBC8WxXJcpa+HVXmJCoOklGnDrRlHJ/umYpfjvb4P/1855tEj+fu1tjELkiXOEcRCP8U2sASFLtIRpL8bpIL3LcNVceYDxo7Jhku/1ibwzvQJ52Tkwpvm+cowfBWxmhuQym9060stT89Z47hz7oZRqgPV1whf1dPub1fxMRwj9BpSmG9pz9AdCjPjAUPOpUbW1CTqfo/tbU8Pc=
2021-07-28T09:32:15.931 INFO:teuthology.kill:  smithi164.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7o4kXwBmmH+EzLzPLY1ylHhgSvyYFSGj/xIKon/+Z9PN0yLYL1rA3pYFpDegVW3gcvmm1oigY/Pjum/R8OOuOWl6yAXVcwRhcJyTqUSbCzTSCVToUCFXQrXQDT6g4aPKD2DiYpQJVrgiNUYsQQuorFwn88hPJMFaxmWoUpolGu5NY8onaqEOc+wkxyORo1q+tTfAEOUtTH4/mauKzlvZBdeBZfsrT8EsFKvX8SObsOGIb9ut9pVVm4MU8Q9CSte3dLP1FEmlLD0R0ncK7dF4qtnpQ8kWBYj8nfNZJ2iAQ9n+ziMRhBH82Rs2sKKqyRNSHxpatu4HOynbX0uR5jw9vYlSc2+PmKvPXkNp44pKHxQtnRoA1DQaMpw7TMwrNUPpGCSIyUH8nZmSmC6AMv4rPHHLmb0f7cdfiS5NNeBViZHwNhHv0isyypxfm5afG1oaxHfScAyn51NnEDz0Nqcr2AoOB3UoExFz3mXtmbo51/tzARvsd10iOSt4vYPKeCPM=
2021-07-28T09:32:15.931 INFO:teuthology.kill:2021-07-28 09:32:15,929.929 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-07-28T09:32:15.933 INFO:teuthology.kill:2021-07-28 09:32:15,932.932 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-07-28T09:32:16.008 INFO:teuthology.kill:2021-07-28 09:32:16,007.007 INFO:teuthology.orchestra.console:Power off smithi031
2021-07-28T09:32:16.023 INFO:teuthology.kill:2021-07-28 09:32:16,023.023 INFO:teuthology.orchestra.console:Power off smithi164
2021-07-28T09:32:24.382 INFO:teuthology.kill:2021-07-28 09:32:24,381.381 INFO:teuthology.orchestra.console:Power off for smithi031 completed
2021-07-28T09:32:24.388 INFO:teuthology.kill:2021-07-28 09:32:24,388.388 INFO:teuthology.orchestra.console:Power off for smithi164 completed
2021-07-28T09:32:24.629 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
2021-07-28T09:32:24.629 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
2021-07-28T09:32:24.630 INFO:teuthology.misc:Compressing logs...
2021-07-28T09:32:24.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-07-28T09:33:24.691 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/teuthology/orchestra/cluster.py", line 64, in <listcomp>
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/teuthology/orchestra/remote.py", line 509, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_73aa7e3b960c7ffac669297b6aa86606265edd1b/teuthology/orchestra/run.py", line 434, in run
    transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-07-28T09:33:24.754 INFO:teuthology.kill:No teuthology processes running
2021-07-28T09:33:25.869 INFO:teuthology.kill:Nuking machines: ['smithi031', 'smithi164']
2021-07-28T09:33:26.814 INFO:teuthology.kill:2021-07-28 09:33:26,814.814 INFO:teuthology.nuke:targets:
2021-07-28T09:33:26.815 INFO:teuthology.kill:  smithi031.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbPKoeaQzW9qqtQ2uLy1RT0aGI8pmfrPxat/BcPyb7vg+Ahglizbkv5FO8G9H7sS1IgWyxKUA48Qdlp4pZzlHQ6aAMLQX1FFao5vpRr1m9u5BPQ4Dk/cF1AyWLV8C6aZozAwdZ5MMpMewGQtN4JYJ8673Bn+DdVBmYIapFl9WSRIcxtjjCfi1YDZQmJ0azL55ymgLFUcIIom1obF9j2Y542UITWodLIBa6m/9XgnCBRf6Y+BBaKIo4Xt1a6/7JBrk5NWengWOl8EXRNU5lnJUr0qvl5doG3fJs/ECS67DWg63XgMKmvHBC8WxXJcpa+HVXmJCoOklGnDrRlHJ/umYpfjvb4P/1855tEj+fu1tjELkiXOEcRCP8U2sASFLtIRpL8bpIL3LcNVceYDxo7Jhku/1ibwzvQJ52Tkwpvm+cowfBWxmhuQym9060stT89Z47hz7oZRqgPV1whf1dPub1fxMRwj9BpSmG9pz9AdCjPjAUPOpUbW1CTqfo/tbU8Pc=
2021-07-28T09:33:26.815 INFO:teuthology.kill:  smithi164.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7o4kXwBmmH+EzLzPLY1ylHhgSvyYFSGj/xIKon/+Z9PN0yLYL1rA3pYFpDegVW3gcvmm1oigY/Pjum/R8OOuOWl6yAXVcwRhcJyTqUSbCzTSCVToUCFXQrXQDT6g4aPKD2DiYpQJVrgiNUYsQQuorFwn88hPJMFaxmWoUpolGu5NY8onaqEOc+wkxyORo1q+tTfAEOUtTH4/mauKzlvZBdeBZfsrT8EsFKvX8SObsOGIb9ut9pVVm4MU8Q9CSte3dLP1FEmlLD0R0ncK7dF4qtnpQ8kWBYj8nfNZJ2iAQ9n+ziMRhBH82Rs2sKKqyRNSHxpatu4HOynbX0uR5jw9vYlSc2+PmKvPXkNp44pKHxQtnRoA1DQaMpw7TMwrNUPpGCSIyUH8nZmSmC6AMv4rPHHLmb0f7cdfiS5NNeBViZHwNhHv0isyypxfm5afG1oaxHfScAyn51NnEDz0Nqcr2AoOB3UoExFz3mXtmbo51/tzARvsd10iOSt4vYPKeCPM=
2021-07-28T09:33:26.835 INFO:teuthology.kill:2021-07-28 09:33:26,834.834 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-07-28T09:33:26.838 INFO:teuthology.kill:2021-07-28 09:33:26,838.838 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-07-28T09:33:26.893 INFO:teuthology.kill:2021-07-28 09:33:26,893.893 INFO:teuthology.orchestra.console:Power off smithi031
2021-07-28T09:33:26.910 INFO:teuthology.kill:2021-07-28 09:33:26,909.909 INFO:teuthology.orchestra.console:Power off smithi164
2021-07-28T09:33:27.046 INFO:teuthology.kill:2021-07-28 09:33:27,046.046 INFO:teuthology.orchestra.console:Power off for smithi031 completed
2021-07-28T09:33:27.057 INFO:teuthology.kill:2021-07-28 09:33:27,056.056 INFO:teuthology.orchestra.console:Power off for smithi164 completed
2021-07-28T09:33:27.191 INFO:teuthology.kill:2021-07-28 09:33:27,189.189 INFO:teuthology.lock.ops:unlocked smithi031.front.sepia.ceph.com
2021-07-28T09:33:27.202 INFO:teuthology.kill:2021-07-28 09:33:27,202.202 INFO:teuthology.lock.ops:unlocked smithi164.front.sepia.ceph.com
2021-07-28T09:35:27.498 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-07-28T09:35:27.617 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2021-07-28T09:35:27.618 INFO:teuthology.nuke:Checking targets against current locks
2021-07-28T09:35:27.691 INFO:teuthology.nuke:Not nuking smithi164.front.sepia.ceph.com because description doesn't match
2021-07-28T09:35:27.692 INFO:teuthology.nuke:Not nuking smithi031.front.sepia.ceph.com because description doesn't match