2021-06-28T21:53:41.108 INFO:root:teuthology version: 1.1.0-26631ede 2021-06-28T21:53:41.244 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi064.front.sepia.ceph.com' reimaging 2021-06-28T21:53:41.245 INFO:teuthology.lock.ops:Updating [ubuntu@smithi064.front.sepia.ceph.com]: reset os type and version on server 2021-06-28T21:53:41.246 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2021-06-28T21:53:41.264 INFO:teuthology.lock.ops:Node 'ubuntu@smithi064.front.sepia.ceph.com' reimaging is complete 2021-06-28T21:53:41.265 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi159.front.sepia.ceph.com' reimaging 2021-06-28T21:53:41.265 INFO:teuthology.lock.ops:Updating [ubuntu@smithi159.front.sepia.ceph.com]: reset os type and version on server 2021-06-28T21:53:41.265 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2021-06-28T21:53:41.287 INFO:teuthology.lock.ops:Node 'ubuntu@smithi159.front.sepia.ceph.com' reimaging is complete 2021-06-28T21:53:41.485 INFO:teuthology.provision.fog.smithi064:Scheduling deploy of ubuntu 20.04 2021-06-28T21:53:41.490 INFO:teuthology.provision.fog.smithi159:Scheduling deploy of ubuntu 20.04 2021-06-28T21:53:41.889 INFO:teuthology.orchestra.console:Power off smithi159 2021-06-28T21:53:41.936 INFO:teuthology.orchestra.console:Power off smithi064 2021-06-28T21:53:54.389 INFO:teuthology.orchestra.console:Power off for smithi159 completed 2021-06-28T21:53:54.420 INFO:teuthology.orchestra.console:Power off for smithi064 completed 2021-06-28T21:53:54.494 INFO:teuthology.orchestra.console:Power on smithi159 2021-06-28T21:53:54.527 INFO:teuthology.orchestra.console:Power on smithi064 2021-06-28T21:53:58.785 INFO:teuthology.orchestra.console:Power on for smithi064 completed 2021-06-28T21:53:58.888 INFO:teuthology.provision.fog.smithi064:Waiting for deploy to finish 2021-06-28T21:54:02.866 INFO:teuthology.orchestra.console:Power on for smithi159 completed 2021-06-28T21:54:02.967 INFO:teuthology.provision.fog.smithi159:Waiting for deploy to finish 2021-06-28T21:59:05.498 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-28T21:59:05.956 INFO:teuthology.provision.fog.smithi064:Node is ready 2021-06-28T21:59:05.963 INFO:teuthology.orchestra.run.smithi064.stdout:smithi064.front.sepia.ceph.com 2021-06-28T21:59:05.970 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64 smithi064.front.sepia.ceph.com smithi064 2021-06-28T21:59:06.162 INFO:teuthology.provision.fog.smithi064:Deploy complete! 2021-06-28T21:59:15.509 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-28T21:59:15.976 INFO:teuthology.provision.fog.smithi159:Node is ready 2021-06-28T21:59:15.982 INFO:teuthology.orchestra.run.smithi159.stdout:smithi159.front.sepia.ceph.com 2021-06-28T21:59:15.989 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.15.159 smithi159.front.sepia.ceph.com smithi159 2021-06-28T21:59:16.163 INFO:teuthology.provision.fog.smithi159:Deploy complete! 2021-06-28T21:59:16.324 INFO:teuthology.lock.ops:Checking smithi064.front.sepia.ceph.com 2021-06-28T21:59:16.325 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-28T21:59:16.343 INFO:teuthology.lock.ops:Checking smithi159.front.sepia.ceph.com 2021-06-28T21:59:16.343 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-28T21:59:16.362 INFO:teuthology.lock.ops:Updating [smithi064.front.sepia.ceph.com]: set os type and version on server 2021-06-28T21:59:16.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-28T21:59:16.546 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2021-06-28T21:59:16.597 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="Ubuntu" 2021-06-28T21:59:16.597 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-06-28T21:59:16.598 INFO:teuthology.orchestra.run.smithi064.stdout:ID=ubuntu 2021-06-28T21:59:16.598 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE=debian 2021-06-28T21:59:16.599 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-06-28T21:59:16.599 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="20.04" 2021-06-28T21:59:16.599 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://www.ubuntu.com/" 2021-06-28T21:59:16.600 INFO:teuthology.orchestra.run.smithi064.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-06-28T21:59:16.600 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-06-28T21:59:16.600 INFO:teuthology.orchestra.run.smithi064.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-06-28T21:59:16.601 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_CODENAME=focal 2021-06-28T21:59:16.601 INFO:teuthology.orchestra.run.smithi064.stdout:UBUNTU_CODENAME=focal 2021-06-28T21:59:16.603 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2021-06-28T21:59:16.619 INFO:teuthology.lock.ops:Updating [smithi159.front.sepia.ceph.com]: set os type and version on server 2021-06-28T21:59:16.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-28T21:59:16.835 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64 2021-06-28T21:59:16.842 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="Ubuntu" 2021-06-28T21:59:16.843 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-06-28T21:59:16.843 INFO:teuthology.orchestra.run.smithi159.stdout:ID=ubuntu 2021-06-28T21:59:16.843 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE=debian 2021-06-28T21:59:16.844 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-06-28T21:59:16.845 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="20.04" 2021-06-28T21:59:16.845 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://www.ubuntu.com/" 2021-06-28T21:59:16.845 INFO:teuthology.orchestra.run.smithi159.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-06-28T21:59:16.845 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-06-28T21:59:16.846 INFO:teuthology.orchestra.run.smithi159.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-06-28T21:59:16.846 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_CODENAME=focal 2021-06-28T21:59:16.846 INFO:teuthology.orchestra.run.smithi159.stdout:UBUNTU_CODENAME=focal 2021-06-28T21:59:16.847 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2021-06-28T21:59:16.939 INFO:teuthology.dispatcher.supervisor:Running job 6238902 2021-06-28T21:59:16.940 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/virtualenv/bin/teuthology -v --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2021-06-27_15:04:00-rados-wip-sage3-testing-2021-06-26-1419-distro-basic-smithi/6238902 --name sage-2021-06-27_15:04:00-rados-wip-sage3-testing-2021-06-26-1419-distro-basic-smithi --description rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/snaps-few-objects} -- /home/teuthworker/archive/sage-2021-06-27_15:04:00-rados-wip-sage3-testing-2021-06-26-1419-distro-basic-smithi/6238902/orig.config.yaml 2021-06-28T21:59:16.945 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/sage-2021-06-27_15:04:00-rados-wip-sage3-testing-2021-06-26-1419-distro-basic-smithi/6238902 2021-06-28T21:59:16.945 INFO:teuthology.dispatcher.supervisor:Job PID: 12824 2021-06-28T21:59:16.946 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-06-29T10:00:08.446 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-06-29T10:00:08.821 INFO:teuthology.kill:Killing Pids: {12824} 2021-06-29T10:00:10.562 INFO:teuthology.kill:Nuking machines: ['smithi064', 'smithi159'] 2021-06-29T10:00:12.924 INFO:teuthology.kill:2021-06-29 10:00:12,918.918 INFO:teuthology.nuke:targets: 2021-06-29T10:00:12.925 INFO:teuthology.kill: smithi064.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD1XoTbXC2OkuAfft3ngv6ZiHQtmtJoZejuOiNtt7pJWT+Fnb1KvQBCL/skcSBQpHNiHCfx5yjzI9SHZw7emILzLp6k/eYBORHTKqGD1jw6jW+Thccd0zXSxlwZsXulHZnJMx/r+RniYmP7h68CIuBEeKlU3Zps4XZ6LeKjJQIc2RqSiWgQOdhDv0t7OHdIJfGmrQEG6MZATXyIbNCx2Oewh8P115piqpI2ucEahkGq0nFj+x3FiWwaJUfRA5lbAshRBdg+4jXihKUiNq0w2eyPVb0G07040WP+S55HGQI4rRlyIHHz3FvzOTU7B/v/Dv7wIEm5tLzX9ur2X9u/Mhaked20JtLpLDLvxffcJVg3GN4WFRZ0n8PNVYD20znRDIwPhqSrDJHbrlRlUhUYCkia+TwNkndLhHbiaHEs7/GvEwEzT9X+Ct/LfvdweFvpuWhUk6aJBqgMmdVH8sGY+Pc8qbyhByHhVxUfy2W+1h4qYfJKRZEci3UBGbqzus35Km0= 2021-06-29T10:00:12.925 INFO:teuthology.kill: smithi159.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtjMsSY1b/vM0h2ye1y6xj2U8trBO40cELb9ubaxGX5VUtJc4OrPmqEFXyJCWpwefTuqiV6r4EkICyXOxE2LdKNr15PEbRQdO2hdYWIf85cRPBrAHBYFnXz2r2ByoT3bt3+yiHzHI4gzAJkxTvetRtyx8u/N2QsApssL4FnSzR4hdKiDwqZQJLz43FP3vmRQSo56AvGsVZMpTVYYrRYuxBKcoOLd3n2tu7HX/QGY3GAXKMt6LqhDnGXuQDCt6uJtl0/hERG2GKoTPPhzSdpZ2fmInI6GQ1NlK8kWQnjPepCX8BPUpMOsh+1Aixv5F+CasXVCS/SU5a6E6Ed1pFc/UzREPvxGl5vsVSVM7UTXAEQi/X3YLAqPuk6RsegF6XJ20kgQMXb03r/4zfljA1zmin31tq5TDvTkSvvhzcmRhh5+PuT2zRQVbWdOI3IowW61yZi/wDRBtn+nlMEIzuqDVXq9oRZU6o/5y7NKpF90yLse71/p85CRRUbJohjk/Y3dM= 2021-06-29T10:00:12.926 INFO:teuthology.kill:2021-06-29 10:00:12,924.924 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-29T10:00:12.929 INFO:teuthology.kill:2021-06-29 10:00:12,929.929 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-29T10:00:13.047 INFO:teuthology.kill:2021-06-29 10:00:13,044.044 INFO:teuthology.orchestra.console:Power off smithi064 2021-06-29T10:00:13.088 INFO:teuthology.kill:2021-06-29 10:00:13,088.088 INFO:teuthology.orchestra.console:Power off smithi159 2021-06-29T10:00:21.550 INFO:teuthology.kill:2021-06-29 10:00:21,546.546 INFO:teuthology.orchestra.console:Power off for smithi159 completed 2021-06-29T10:00:25.710 INFO:teuthology.kill:2021-06-29 10:00:25,702.702 INFO:teuthology.orchestra.console:Power off for smithi064 completed 2021-06-29T10:00:26.180 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'] 2021-06-29T10:00:26.180 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2021-06-29T10:00:26.181 INFO:teuthology.misc:Compressing logs... 2021-06-29T10:00:26.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-29T10:01:26.255 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-06-29T10:01:26.381 INFO:teuthology.kill:No teuthology processes running 2021-06-29T10:01:27.876 INFO:teuthology.kill:Nuking machines: ['smithi064', 'smithi159'] 2021-06-29T10:01:28.906 INFO:teuthology.kill:2021-06-29 10:01:28,906.906 INFO:teuthology.nuke:targets: 2021-06-29T10:01:28.907 INFO:teuthology.kill: smithi064.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD1XoTbXC2OkuAfft3ngv6ZiHQtmtJoZejuOiNtt7pJWT+Fnb1KvQBCL/skcSBQpHNiHCfx5yjzI9SHZw7emILzLp6k/eYBORHTKqGD1jw6jW+Thccd0zXSxlwZsXulHZnJMx/r+RniYmP7h68CIuBEeKlU3Zps4XZ6LeKjJQIc2RqSiWgQOdhDv0t7OHdIJfGmrQEG6MZATXyIbNCx2Oewh8P115piqpI2ucEahkGq0nFj+x3FiWwaJUfRA5lbAshRBdg+4jXihKUiNq0w2eyPVb0G07040WP+S55HGQI4rRlyIHHz3FvzOTU7B/v/Dv7wIEm5tLzX9ur2X9u/Mhaked20JtLpLDLvxffcJVg3GN4WFRZ0n8PNVYD20znRDIwPhqSrDJHbrlRlUhUYCkia+TwNkndLhHbiaHEs7/GvEwEzT9X+Ct/LfvdweFvpuWhUk6aJBqgMmdVH8sGY+Pc8qbyhByHhVxUfy2W+1h4qYfJKRZEci3UBGbqzus35Km0= 2021-06-29T10:01:28.907 INFO:teuthology.kill: smithi159.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtjMsSY1b/vM0h2ye1y6xj2U8trBO40cELb9ubaxGX5VUtJc4OrPmqEFXyJCWpwefTuqiV6r4EkICyXOxE2LdKNr15PEbRQdO2hdYWIf85cRPBrAHBYFnXz2r2ByoT3bt3+yiHzHI4gzAJkxTvetRtyx8u/N2QsApssL4FnSzR4hdKiDwqZQJLz43FP3vmRQSo56AvGsVZMpTVYYrRYuxBKcoOLd3n2tu7HX/QGY3GAXKMt6LqhDnGXuQDCt6uJtl0/hERG2GKoTPPhzSdpZ2fmInI6GQ1NlK8kWQnjPepCX8BPUpMOsh+1Aixv5F+CasXVCS/SU5a6E6Ed1pFc/UzREPvxGl5vsVSVM7UTXAEQi/X3YLAqPuk6RsegF6XJ20kgQMXb03r/4zfljA1zmin31tq5TDvTkSvvhzcmRhh5+PuT2zRQVbWdOI3IowW61yZi/wDRBtn+nlMEIzuqDVXq9oRZU6o/5y7NKpF90yLse71/p85CRRUbJohjk/Y3dM= 2021-06-29T10:01:28.929 INFO:teuthology.kill:2021-06-29 10:01:28,929.929 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-29T10:01:28.944 INFO:teuthology.kill:2021-06-29 10:01:28,944.944 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-29T10:01:29.023 INFO:teuthology.kill:2021-06-29 10:01:29,023.023 INFO:teuthology.orchestra.console:Power off smithi159 2021-06-29T10:01:29.111 INFO:teuthology.kill:2021-06-29 10:01:29,111.111 INFO:teuthology.orchestra.console:Power off smithi064 2021-06-29T10:01:29.189 INFO:teuthology.kill:2021-06-29 10:01:29,189.189 INFO:teuthology.orchestra.console:Power off for smithi159 completed 2021-06-29T10:01:29.261 INFO:teuthology.kill:2021-06-29 10:01:29,260.260 INFO:teuthology.orchestra.console:Power off for smithi064 completed 2021-06-29T10:01:29.334 INFO:teuthology.kill:2021-06-29 10:01:29,333.333 INFO:teuthology.lock.ops:unlocked smithi159.front.sepia.ceph.com 2021-06-29T10:01:29.414 INFO:teuthology.kill:2021-06-29 10:01:29,413.413 INFO:teuthology.lock.ops:unlocked smithi064.front.sepia.ceph.com 2021-06-29T10:03:29.738 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-06-29T10:03:29.829 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-06-29T10:03:29.830 INFO:teuthology.nuke:Checking targets against current locks 2021-06-29T10:03:29.916 INFO:teuthology.nuke:Not nuking smithi064.front.sepia.ceph.com because description doesn't match 2021-06-29T10:03:29.917 INFO:teuthology.nuke:Not nuking smithi159.front.sepia.ceph.com because description doesn't match