2021-05-05T16:51:02.361 INFO:root:teuthology version: 1.1.0-19220a3b 2021-05-05T16:51:03.498 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba005.front.sepia.ceph.com' reimaging 2021-05-05T16:51:03.499 INFO:teuthology.lock.ops:Updating [ubuntu@gibba005.front.sepia.ceph.com]: reset os type and version on server 2021-05-05T16:51:03.500 INFO:teuthology.lock.ops:Updating gibba005.front.sepia.ceph.com on lock server 2021-05-05T16:51:03.515 INFO:teuthology.lock.ops:Node 'ubuntu@gibba005.front.sepia.ceph.com' reimaging is complete 2021-05-05T16:51:03.516 INFO:teuthology.lock.ops:Start node 'ubuntu@gibba025.front.sepia.ceph.com' reimaging 2021-05-05T16:51:03.516 INFO:teuthology.lock.ops:Updating [ubuntu@gibba025.front.sepia.ceph.com]: reset os type and version on server 2021-05-05T16:51:03.516 INFO:teuthology.lock.ops:Updating gibba025.front.sepia.ceph.com on lock server 2021-05-05T16:51:03.532 INFO:teuthology.lock.ops:Node 'ubuntu@gibba025.front.sepia.ceph.com' reimaging is complete 2021-05-05T16:51:03.718 INFO:teuthology.provision.fog.gibba005:Scheduling deploy of ubuntu 20.04 2021-05-05T16:51:03.731 INFO:teuthology.provision.fog.gibba025:Scheduling deploy of ubuntu 20.04 2021-05-05T16:51:04.018 INFO:teuthology.orchestra.console:Power off gibba005 2021-05-05T16:51:04.032 INFO:teuthology.orchestra.console:Power off gibba025 2021-05-05T16:51:16.525 INFO:teuthology.orchestra.console:Power off for gibba005 completed 2021-05-05T16:51:16.542 INFO:teuthology.orchestra.console:Power off for gibba025 completed 2021-05-05T16:51:16.626 INFO:teuthology.orchestra.console:Power on gibba005 2021-05-05T16:51:16.643 INFO:teuthology.orchestra.console:Power on gibba025 2021-05-05T16:51:20.985 INFO:teuthology.orchestra.console:Power on for gibba025 completed 2021-05-05T16:51:21.087 INFO:teuthology.provision.fog.gibba025:Waiting for deploy to finish 2021-05-05T16:51:25.016 INFO:teuthology.orchestra.console:Power on for gibba005 completed 2021-05-05T16:51:25.117 INFO:teuthology.provision.fog.gibba005:Waiting for deploy to finish 2021-05-05T16:53:56.148 ERROR:teuthology.orchestra.connection:Error authenticating with gibba005.front.sepia.ceph.com: Authentication failed. 2021-05-05T16:56:09.199 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-05T16:56:09.680 INFO:teuthology.provision.fog.gibba025:Node is ready 2021-05-05T16:56:09.688 INFO:teuthology.orchestra.run.gibba025.stdout:gibba025.front.sepia.ceph.com 2021-05-05T16:56:09.698 INFO:teuthology.orchestra.run.gibba025.stdout:172.21.2.125 gibba025.front.sepia.ceph.com gibba025 2021-05-05T16:56:09.894 INFO:teuthology.provision.fog.gibba025:Deploy complete! 2021-05-05T16:56:10.903 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-05T16:56:11.253 INFO:teuthology.provision.fog.gibba005:Node is ready 2021-05-05T16:56:11.259 INFO:teuthology.orchestra.run.gibba005.stdout:gibba005.front.sepia.ceph.com 2021-05-05T16:56:11.310 INFO:teuthology.orchestra.run.gibba005.stdout:172.21.2.105 gibba005.front.sepia.ceph.com gibba005 2021-05-05T16:56:11.541 INFO:teuthology.provision.fog.gibba005:Deploy complete! 2021-05-05T16:56:11.668 INFO:teuthology.lock.ops:Checking gibba005.front.sepia.ceph.com 2021-05-05T16:56:11.669 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-05T16:56:11.685 INFO:teuthology.lock.ops:Checking gibba025.front.sepia.ceph.com 2021-05-05T16:56:11.686 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-05T16:56:11.704 INFO:teuthology.lock.ops:Updating [gibba005.front.sepia.ceph.com]: set os type and version on server 2021-05-05T16:56:11.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-05T16:56:11.841 INFO:teuthology.orchestra.run.gibba005.stdout:x86_64 2021-05-05T16:56:11.890 INFO:teuthology.orchestra.run.gibba005.stdout:NAME="Ubuntu" 2021-05-05T16:56:11.890 INFO:teuthology.orchestra.run.gibba005.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-05-05T16:56:11.890 INFO:teuthology.orchestra.run.gibba005.stdout:ID=ubuntu 2021-05-05T16:56:11.891 INFO:teuthology.orchestra.run.gibba005.stdout:ID_LIKE=debian 2021-05-05T16:56:11.891 INFO:teuthology.orchestra.run.gibba005.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-05-05T16:56:11.891 INFO:teuthology.orchestra.run.gibba005.stdout:VERSION_ID="20.04" 2021-05-05T16:56:11.891 INFO:teuthology.orchestra.run.gibba005.stdout:HOME_URL="https://www.ubuntu.com/" 2021-05-05T16:56:11.892 INFO:teuthology.orchestra.run.gibba005.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-05-05T16:56:11.892 INFO:teuthology.orchestra.run.gibba005.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-05-05T16:56:11.892 INFO:teuthology.orchestra.run.gibba005.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-05-05T16:56:11.892 INFO:teuthology.orchestra.run.gibba005.stdout:VERSION_CODENAME=focal 2021-05-05T16:56:11.893 INFO:teuthology.orchestra.run.gibba005.stdout:UBUNTU_CODENAME=focal 2021-05-05T16:56:11.894 INFO:teuthology.lock.ops:Updating gibba005.front.sepia.ceph.com on lock server 2021-05-05T16:56:11.910 INFO:teuthology.lock.ops:Updating [gibba025.front.sepia.ceph.com]: set os type and version on server 2021-05-05T16:56:11.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-05T16:56:12.071 INFO:teuthology.orchestra.run.gibba025.stdout:x86_64 2021-05-05T16:56:12.079 INFO:teuthology.orchestra.run.gibba025.stdout:NAME="Ubuntu" 2021-05-05T16:56:12.079 INFO:teuthology.orchestra.run.gibba025.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-05-05T16:56:12.079 INFO:teuthology.orchestra.run.gibba025.stdout:ID=ubuntu 2021-05-05T16:56:12.080 INFO:teuthology.orchestra.run.gibba025.stdout:ID_LIKE=debian 2021-05-05T16:56:12.080 INFO:teuthology.orchestra.run.gibba025.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-05-05T16:56:12.080 INFO:teuthology.orchestra.run.gibba025.stdout:VERSION_ID="20.04" 2021-05-05T16:56:12.080 INFO:teuthology.orchestra.run.gibba025.stdout:HOME_URL="https://www.ubuntu.com/" 2021-05-05T16:56:12.081 INFO:teuthology.orchestra.run.gibba025.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-05-05T16:56:12.081 INFO:teuthology.orchestra.run.gibba025.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-05-05T16:56:12.081 INFO:teuthology.orchestra.run.gibba025.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-05-05T16:56:12.081 INFO:teuthology.orchestra.run.gibba025.stdout:VERSION_CODENAME=focal 2021-05-05T16:56:12.081 INFO:teuthology.orchestra.run.gibba025.stdout:UBUNTU_CODENAME=focal 2021-05-05T16:56:12.082 INFO:teuthology.lock.ops:Updating gibba025.front.sepia.ceph.com on lock server 2021-05-05T16:56:12.180 INFO:teuthology.dispatcher.supervisor:Running job 6099500 2021-05-05T16:56:12.180 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2021-05-05_16:36:42-rados-benhanokh-NCB-distro-basic-gibba/6099500 --name benhanokh-2021-05-05_16:36:42-rados-benhanokh-NCB-distro-basic-gibba --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} -- /home/teuthworker/archive/benhanokh-2021-05-05_16:36:42-rados-benhanokh-NCB-distro-basic-gibba/6099500/orig.config.yaml 2021-05-05T16:56:12.184 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2021-05-05_16:36:42-rados-benhanokh-NCB-distro-basic-gibba/6099500 2021-05-05T16:56:12.185 INFO:teuthology.dispatcher.supervisor:Job PID: 26788 2021-05-05T16:56:12.185 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-06T04:56:31.789 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-06T04:56:32.016 INFO:teuthology.kill:Killing Pids: {26788} 2021-05-06T04:56:33.163 INFO:teuthology.kill:Nuking machines: ['gibba005', 'gibba025'] 2021-05-06T04:56:34.134 INFO:teuthology.kill:2021-05-06 04:56:34,133.133 INFO:teuthology.nuke:targets: 2021-05-06T04:56:34.135 INFO:teuthology.kill: gibba005.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD6z3wkZfKwsxJtmJosTV3JOlheT5HqUEPo4Poowu+VvdGq5knclmjGrojCwEOrKk93dDtrkYUdMewRaLgmvntcbvbqHdrFDljdD2VQYDO3yijHrHEtdtD7a3Slx+ATuNxlRlT9/2zrs2FS+pMb+QsNGBJOxthSxH0m3RVmJNhBGipZx/bcvf2VwaauYGNMfn8H0ax0EvfyYb+8u9gYNuvFw3NYd5Bp5LF17ImZaOsWk74bqHFhP1G3vvXMkwSV3LCpKfLkrzP7XPqTnS8LRI+tyZ7kjktd1ioeFtjvxvyNRlrrYcAeOlR7Kl/7IePVth4INja9C04cI4F+ykWctXsE74Cb2PLAGDzHH/WU6uq/HVRti8FAKv5lA+d0gmCTOSDIc7toPco5v5YORV7cAbEOyK1Bd99pcjvb+skgiFBwkfMgtR+BCB/MpefFzPsHTXBpKMxGE/0VIoiCmc8Ygl1kVCeAYjwD8OSLRcMAHdJeXNVE7HWKgctpG7s48v7sVU0= 2021-05-06T04:56:34.135 INFO:teuthology.kill: gibba025.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6EHumFiI56JK+voe/6wt4Wx0fd3+V5vGBfJpsFERtrWIz6AbGNyADX6zh7/YmjGrIP2f1arm8mlstvLLvTvEpof2zwMrZBM2GIaCiUNsG43df108+9lzNi6smUyQP1elVKut9wMIEWVwdfJADCoSF88HJqiRYoR8Md4kMCy4rq5cKreqKCjEdNA9QdlVZ3pCqKBHIJ2lYVf+ZteDvJnlkd6acMM9un51kW5HV3U01/e/BOpwSIf6YyNrYD2/xwOe1qgnpNI21/uL1Z6wyM5Xpw5aU7Ft6UVdjTQP3HwEcz2tr3HFPhBoamupf8UFq03prr2E0Qn1fRCTlnQK184Xm1hFn+KO+WfQk4tDQS4osVAsoGHcMcXiz/57vanxoFMDnR0P3KmZIOXkSKMTDe0kbWoJ7idYZIT5PkT9aVRmEV8JdjvqpPnUSkyfKMrF8X5bJZIbRi+Og89t1OAVCjC3No6xkvY97+rRHoAywM/txsQez7LD6R0t62flbHiOKdFc= 2021-05-06T04:56:34.135 INFO:teuthology.kill:2021-05-06 04:56:34,134.134 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-06T04:56:34.137 INFO:teuthology.kill:2021-05-06 04:56:34,136.136 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-06T04:56:34.200 INFO:teuthology.kill:2021-05-06 04:56:34,200.200 INFO:teuthology.orchestra.console:Power off gibba025 2021-05-06T04:56:34.242 INFO:teuthology.kill:2021-05-06 04:56:34,240.240 INFO:teuthology.orchestra.console:Power off gibba005 2021-05-06T04:56:42.621 INFO:teuthology.kill:2021-05-06 04:56:42,620.620 INFO:teuthology.orchestra.console:Power off for gibba005 completed 2021-05-06T04:56:46.741 INFO:teuthology.kill:2021-05-06 04:56:46,740.740 INFO:teuthology.orchestra.console:Power off for gibba025 completed 2021-05-06T04:56:47.035 INFO:teuthology.task.internal:roles: ubuntu@gibba005.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-05-06T04:56:47.038 INFO:teuthology.task.internal:roles: ubuntu@gibba025.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-05-06T04:56:47.038 INFO:teuthology.misc:Compressing logs... 2021-05-06T04:56:47.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-06T04:57:47.044 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_19220a3bd6e252c6e8260827019668a766d85490/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-06T04:57:47.114 INFO:teuthology.kill:No teuthology processes running 2021-05-06T04:57:48.494 INFO:teuthology.kill:Nuking machines: ['gibba005', 'gibba025'] 2021-05-06T04:57:49.509 INFO:teuthology.kill:2021-05-06 04:57:49,509.509 INFO:teuthology.nuke:targets: 2021-05-06T04:57:49.510 INFO:teuthology.kill: gibba005.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD6z3wkZfKwsxJtmJosTV3JOlheT5HqUEPo4Poowu+VvdGq5knclmjGrojCwEOrKk93dDtrkYUdMewRaLgmvntcbvbqHdrFDljdD2VQYDO3yijHrHEtdtD7a3Slx+ATuNxlRlT9/2zrs2FS+pMb+QsNGBJOxthSxH0m3RVmJNhBGipZx/bcvf2VwaauYGNMfn8H0ax0EvfyYb+8u9gYNuvFw3NYd5Bp5LF17ImZaOsWk74bqHFhP1G3vvXMkwSV3LCpKfLkrzP7XPqTnS8LRI+tyZ7kjktd1ioeFtjvxvyNRlrrYcAeOlR7Kl/7IePVth4INja9C04cI4F+ykWctXsE74Cb2PLAGDzHH/WU6uq/HVRti8FAKv5lA+d0gmCTOSDIc7toPco5v5YORV7cAbEOyK1Bd99pcjvb+skgiFBwkfMgtR+BCB/MpefFzPsHTXBpKMxGE/0VIoiCmc8Ygl1kVCeAYjwD8OSLRcMAHdJeXNVE7HWKgctpG7s48v7sVU0= 2021-05-06T04:57:49.511 INFO:teuthology.kill: gibba025.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6EHumFiI56JK+voe/6wt4Wx0fd3+V5vGBfJpsFERtrWIz6AbGNyADX6zh7/YmjGrIP2f1arm8mlstvLLvTvEpof2zwMrZBM2GIaCiUNsG43df108+9lzNi6smUyQP1elVKut9wMIEWVwdfJADCoSF88HJqiRYoR8Md4kMCy4rq5cKreqKCjEdNA9QdlVZ3pCqKBHIJ2lYVf+ZteDvJnlkd6acMM9un51kW5HV3U01/e/BOpwSIf6YyNrYD2/xwOe1qgnpNI21/uL1Z6wyM5Xpw5aU7Ft6UVdjTQP3HwEcz2tr3HFPhBoamupf8UFq03prr2E0Qn1fRCTlnQK184Xm1hFn+KO+WfQk4tDQS4osVAsoGHcMcXiz/57vanxoFMDnR0P3KmZIOXkSKMTDe0kbWoJ7idYZIT5PkT9aVRmEV8JdjvqpPnUSkyfKMrF8X5bJZIbRi+Og89t1OAVCjC3No6xkvY97+rRHoAywM/txsQez7LD6R0t62flbHiOKdFc= 2021-05-06T04:57:49.534 INFO:teuthology.kill:2021-05-06 04:57:49,534.534 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-06T04:57:49.537 INFO:teuthology.kill:2021-05-06 04:57:49,537.537 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-06T04:57:49.598 INFO:teuthology.kill:2021-05-06 04:57:49,598.598 INFO:teuthology.orchestra.console:Power off gibba025 2021-05-06T04:57:49.614 INFO:teuthology.kill:2021-05-06 04:57:49,614.614 INFO:teuthology.orchestra.console:Power off gibba005 2021-05-06T04:57:49.749 INFO:teuthology.kill:2021-05-06 04:57:49,749.749 INFO:teuthology.orchestra.console:Power off for gibba025 completed 2021-05-06T04:57:49.766 INFO:teuthology.kill:2021-05-06 04:57:49,765.765 INFO:teuthology.orchestra.console:Power off for gibba005 completed 2021-05-06T04:57:49.891 INFO:teuthology.kill:2021-05-06 04:57:49,890.890 INFO:teuthology.lock.ops:unlocked gibba025.front.sepia.ceph.com 2021-05-06T04:57:49.910 INFO:teuthology.kill:2021-05-06 04:57:49,910.910 INFO:teuthology.lock.ops:unlocked gibba005.front.sepia.ceph.com 2021-05-06T04:59:50.122 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15