2021-07-22T18:14:41.270 INFO:root:teuthology version: 1.1.0-b6f3efab 2021-07-22T18:14:41.632 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi008.front.sepia.ceph.com' reimaging 2021-07-22T18:14:41.633 INFO:teuthology.lock.ops:Updating [ubuntu@smithi008.front.sepia.ceph.com]: reset os type and version on server 2021-07-22T18:14:41.633 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2021-07-22T18:14:41.689 INFO:teuthology.lock.ops:Node 'ubuntu@smithi008.front.sepia.ceph.com' reimaging is complete 2021-07-22T18:14:41.690 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi187.front.sepia.ceph.com' reimaging 2021-07-22T18:14:41.690 INFO:teuthology.lock.ops:Updating [ubuntu@smithi187.front.sepia.ceph.com]: reset os type and version on server 2021-07-22T18:14:41.691 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2021-07-22T18:14:41.713 INFO:teuthology.lock.ops:Node 'ubuntu@smithi187.front.sepia.ceph.com' reimaging is complete 2021-07-22T18:14:41.913 INFO:teuthology.provision.fog.smithi008:Scheduling deploy of ubuntu 20.04 2021-07-22T18:14:41.933 INFO:teuthology.provision.fog.smithi187:Scheduling deploy of ubuntu 20.04 2021-07-22T18:14:42.284 INFO:teuthology.orchestra.console:Power off smithi187 2021-07-22T18:14:42.316 INFO:teuthology.orchestra.console:Power off smithi008 2021-07-22T18:14:50.720 INFO:teuthology.orchestra.console:Power off for smithi008 completed 2021-07-22T18:14:50.826 INFO:teuthology.orchestra.console:Power on smithi008 2021-07-22T18:14:54.846 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2021-07-22T18:14:54.952 INFO:teuthology.orchestra.console:Power on smithi187 2021-07-22T18:14:59.272 INFO:teuthology.orchestra.console:Power on for smithi008 completed 2021-07-22T18:14:59.373 INFO:teuthology.provision.fog.smithi008:Waiting for deploy to finish 2021-07-22T18:15:03.392 INFO:teuthology.orchestra.console:Power on for smithi187 completed 2021-07-22T18:15:03.493 INFO:teuthology.provision.fog.smithi187:Waiting for deploy to finish 2021-07-22T18:17:45.655 ERROR:teuthology.orchestra.connection:Error authenticating with smithi008.front.sepia.ceph.com: Authentication failed. 2021-07-22T18:17:49.761 ERROR:teuthology.orchestra.connection:Error authenticating with smithi187.front.sepia.ceph.com: Authentication failed. 2021-07-22T18:19:52.518 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-07-22T18:19:53.010 INFO:teuthology.provision.fog.smithi187:Node is ready 2021-07-22T18:19:53.018 INFO:teuthology.orchestra.run.smithi187.stdout:smithi187.front.sepia.ceph.com 2021-07-22T18:19:53.069 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.15.187 smithi187.front.sepia.ceph.com smithi187 2021-07-22T18:19:53.320 INFO:teuthology.provision.fog.smithi187:Deploy complete! 2021-07-22T18:19:58.869 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-07-22T18:19:59.378 INFO:teuthology.provision.fog.smithi008:Node is ready 2021-07-22T18:19:59.427 INFO:teuthology.orchestra.run.smithi008.stdout:smithi008.front.sepia.ceph.com 2021-07-22T18:19:59.435 INFO:teuthology.orchestra.run.smithi008.stdout:172.21.15.8 smithi008.front.sepia.ceph.com smithi008 2021-07-22T18:19:59.608 INFO:teuthology.provision.fog.smithi008:Deploy complete! 2021-07-22T18:19:59.768 INFO:teuthology.lock.ops:Checking smithi008.front.sepia.ceph.com 2021-07-22T18:19:59.769 INFO:teuthology.lock.ops:New key found. Updating... 2021-07-22T18:19:59.788 INFO:teuthology.lock.ops:Checking smithi187.front.sepia.ceph.com 2021-07-22T18:19:59.789 INFO:teuthology.lock.ops:New key found. Updating... 2021-07-22T18:19:59.805 INFO:teuthology.lock.ops:Updating [smithi008.front.sepia.ceph.com]: set os type and version on server 2021-07-22T18:19:59.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-22T18:19:59.987 INFO:teuthology.orchestra.run.smithi008.stdout:x86_64 2021-07-22T18:20:00.040 INFO:teuthology.orchestra.run.smithi008.stdout:NAME="Ubuntu" 2021-07-22T18:20:00.040 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-07-22T18:20:00.040 INFO:teuthology.orchestra.run.smithi008.stdout:ID=ubuntu 2021-07-22T18:20:00.041 INFO:teuthology.orchestra.run.smithi008.stdout:ID_LIKE=debian 2021-07-22T18:20:00.041 INFO:teuthology.orchestra.run.smithi008.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-07-22T18:20:00.041 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_ID="20.04" 2021-07-22T18:20:00.041 INFO:teuthology.orchestra.run.smithi008.stdout:HOME_URL="https://www.ubuntu.com/" 2021-07-22T18:20:00.042 INFO:teuthology.orchestra.run.smithi008.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-07-22T18:20:00.042 INFO:teuthology.orchestra.run.smithi008.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-07-22T18:20:00.042 INFO:teuthology.orchestra.run.smithi008.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-07-22T18:20:00.042 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_CODENAME=focal 2021-07-22T18:20:00.043 INFO:teuthology.orchestra.run.smithi008.stdout:UBUNTU_CODENAME=focal 2021-07-22T18:20:00.044 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2021-07-22T18:20:00.060 INFO:teuthology.lock.ops:Updating [smithi187.front.sepia.ceph.com]: set os type and version on server 2021-07-22T18:20:00.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-22T18:20:00.234 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2021-07-22T18:20:00.241 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="Ubuntu" 2021-07-22T18:20:00.241 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-07-22T18:20:00.242 INFO:teuthology.orchestra.run.smithi187.stdout:ID=ubuntu 2021-07-22T18:20:00.242 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE=debian 2021-07-22T18:20:00.242 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-07-22T18:20:00.242 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="20.04" 2021-07-22T18:20:00.243 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://www.ubuntu.com/" 2021-07-22T18:20:00.243 INFO:teuthology.orchestra.run.smithi187.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-07-22T18:20:00.243 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-07-22T18:20:00.243 INFO:teuthology.orchestra.run.smithi187.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-07-22T18:20:00.244 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_CODENAME=focal 2021-07-22T18:20:00.244 INFO:teuthology.orchestra.run.smithi187.stdout:UBUNTU_CODENAME=focal 2021-07-22T18:20:00.244 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2021-07-22T18:20:00.331 INFO:teuthology.dispatcher.supervisor:Running job 6287302 2021-07-22T18:20:00.331 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/virtualenv/bin/teuthology -v --owner scheduled_nojha@teuthology --archive /home/teuthworker/archive/nojha-2021-07-22_18:00:50-rados-wip-sseshasa-testing-2021-07-14-1320-distro-basic-smithi/6287302 --name nojha-2021-07-22_18:00:50-rados-wip-sseshasa-testing-2021-07-14-1320-distro-basic-smithi --description rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/snaps-few-objects} -- /home/teuthworker/archive/nojha-2021-07-22_18:00:50-rados-wip-sseshasa-testing-2021-07-14-1320-distro-basic-smithi/6287302/orig.config.yaml 2021-07-22T18:20:00.335 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/nojha-2021-07-22_18:00:50-rados-wip-sseshasa-testing-2021-07-14-1320-distro-basic-smithi/6287302 2021-07-22T18:20:00.336 INFO:teuthology.dispatcher.supervisor:Job PID: 20421 2021-07-22T18:20:00.336 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-07-23T06:20:18.357 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-07-23T06:20:18.506 INFO:teuthology.kill:Killing Pids: {20421} 2021-07-23T06:20:19.624 INFO:teuthology.kill:Nuking machines: ['smithi008', 'smithi187'] 2021-07-23T06:20:20.526 INFO:teuthology.kill:2021-07-23 06:20:20,525.525 INFO:teuthology.nuke:targets: 2021-07-23T06:20:20.527 INFO:teuthology.kill: smithi008.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNWmTAQ0lAyuTsqSUmfpzfWfU1gALLYx0xProu7ds4cp9hASwPtZPQzh1h3H/bqS5P61p8L/FHOMNSybbc1pBvuAWz+6yKvhT16vizW87Ku30bUFqWbsrfIe8HUSVIr69RyX0DGhZTD/usgTnEuBqMjjMNGpYUI0FD6n4h7P04EwVSqqTwU+Nv+uqgoMPb0JKWNI6Cqxhvub7P1CrPT11ERrs9iNTqkSQgB3ogorlFKRqnXpmGvWuuWjg8VTLQFhn0OaL6xJDGFBXeXwLm0+EqUafthVTOAmAutNenRSSKTvPpjIs+QtVkpskOGoxFvWCDjOkhtTL+SsMWA/ft1TOTRVFjaHMlzFDy/oEd0Z+jP1iz3qZpWvxesZkM/fN9PSuJQXBQIsRMq3jCfPycuaALtcFMQRuruUcP+UMa1RcZwoBgyDc5iTFlYSsFn3Kh+23awtkakMZm50O8NSkUelOi/5R9G6sUal4X+nhmeECVt+ehMXLm//yMfSO+6Sn877E= 2021-07-23T06:20:20.527 INFO:teuthology.kill: smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKCwMomsGFu5DYhsG8017qz6dKIXDx9EWGUtrpvVOhcEFlYE6ezC4wjpf+Kc3XtF+HePyvWPMX1B5RbsVkDukkGt8SvECReD1VMLzkWly6P8Hc3DL9UTEuv/Qj3U4UudaknH4OzEDWvrdXDqdqyNoH+DiGJGHFuO3UxYS8iZtdeC4WfOei0xUwnVhM3uk8hHK/Bi8BqG8aPc/nuPZXrtyOgLk37Jf/821QRo4GCBaP/9L6TbfUO+zAxaOXczfQ8DrbIi+H/l70oeNdJFdKfOcM0YVLQmjzm12jboZJxASi9YjIdKcN0RXxz3fR8S2/LoCSEMdKc/iJ/B3vvl4Xh6MoEYgq9M+A1XFrxmPz03+91jctJKp9kBCixNoutcsjze0mNaJT8R3ZAsGwqs2E9TA897AdPr68Zmp7x0o9wT/muVhabxOnUd+uGeno80XNTJZ/2snl6HohK8ALObqBIdwthwHbiXzpM1Yag5wuC1LlWrIybyvPBnro76ZrsMiWSCM= 2021-07-23T06:20:20.528 INFO:teuthology.kill:2021-07-23 06:20:20,526.526 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-07-23T06:20:20.528 INFO:teuthology.kill:2021-07-23 06:20:20,528.528 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-07-23T06:20:20.583 INFO:teuthology.kill:2021-07-23 06:20:20,583.583 INFO:teuthology.orchestra.console:Power off smithi187 2021-07-23T06:20:20.596 INFO:teuthology.kill:2021-07-23 06:20:20,596.596 INFO:teuthology.orchestra.console:Power off smithi008 2021-07-23T06:20:33.062 INFO:teuthology.kill:2021-07-23 06:20:33,061.061 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2021-07-23T06:20:33.066 INFO:teuthology.kill:2021-07-23 06:20:33,066.066 INFO:teuthology.orchestra.console:Power off for smithi008 completed 2021-07-23T06:20:33.280 INFO:teuthology.task.internal:roles: ubuntu@smithi008.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'] 2021-07-23T06:20:33.281 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2021-07-23T06:20:33.281 INFO:teuthology.misc:Compressing logs... 2021-07-23T06:20:33.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-23T06:20:51.787 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b6f3efab84b70e18e86ec271a07564aeb4ed3bac/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-07-23T06:20:51.842 INFO:teuthology.kill:No teuthology processes running 2021-07-23T06:20:52.916 INFO:teuthology.kill:Nuking machines: ['smithi008', 'smithi187'] 2021-07-23T06:20:53.878 INFO:teuthology.kill:2021-07-23 06:20:53,878.878 INFO:teuthology.nuke:targets: 2021-07-23T06:20:53.879 INFO:teuthology.kill: smithi008.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNWmTAQ0lAyuTsqSUmfpzfWfU1gALLYx0xProu7ds4cp9hASwPtZPQzh1h3H/bqS5P61p8L/FHOMNSybbc1pBvuAWz+6yKvhT16vizW87Ku30bUFqWbsrfIe8HUSVIr69RyX0DGhZTD/usgTnEuBqMjjMNGpYUI0FD6n4h7P04EwVSqqTwU+Nv+uqgoMPb0JKWNI6Cqxhvub7P1CrPT11ERrs9iNTqkSQgB3ogorlFKRqnXpmGvWuuWjg8VTLQFhn0OaL6xJDGFBXeXwLm0+EqUafthVTOAmAutNenRSSKTvPpjIs+QtVkpskOGoxFvWCDjOkhtTL+SsMWA/ft1TOTRVFjaHMlzFDy/oEd0Z+jP1iz3qZpWvxesZkM/fN9PSuJQXBQIsRMq3jCfPycuaALtcFMQRuruUcP+UMa1RcZwoBgyDc5iTFlYSsFn3Kh+23awtkakMZm50O8NSkUelOi/5R9G6sUal4X+nhmeECVt+ehMXLm//yMfSO+6Sn877E= 2021-07-23T06:20:53.880 INFO:teuthology.kill: smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKCwMomsGFu5DYhsG8017qz6dKIXDx9EWGUtrpvVOhcEFlYE6ezC4wjpf+Kc3XtF+HePyvWPMX1B5RbsVkDukkGt8SvECReD1VMLzkWly6P8Hc3DL9UTEuv/Qj3U4UudaknH4OzEDWvrdXDqdqyNoH+DiGJGHFuO3UxYS8iZtdeC4WfOei0xUwnVhM3uk8hHK/Bi8BqG8aPc/nuPZXrtyOgLk37Jf/821QRo4GCBaP/9L6TbfUO+zAxaOXczfQ8DrbIi+H/l70oeNdJFdKfOcM0YVLQmjzm12jboZJxASi9YjIdKcN0RXxz3fR8S2/LoCSEMdKc/iJ/B3vvl4Xh6MoEYgq9M+A1XFrxmPz03+91jctJKp9kBCixNoutcsjze0mNaJT8R3ZAsGwqs2E9TA897AdPr68Zmp7x0o9wT/muVhabxOnUd+uGeno80XNTJZ/2snl6HohK8ALObqBIdwthwHbiXzpM1Yag5wuC1LlWrIybyvPBnro76ZrsMiWSCM= 2021-07-23T06:20:53.899 INFO:teuthology.kill:2021-07-23 06:20:53,898.898 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-07-23T06:20:53.902 INFO:teuthology.kill:2021-07-23 06:20:53,901.901 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-07-23T06:20:53.961 INFO:teuthology.kill:2021-07-23 06:20:53,960.960 INFO:teuthology.orchestra.console:Power off smithi187 2021-07-23T06:20:53.974 INFO:teuthology.kill:2021-07-23 06:20:53,974.974 INFO:teuthology.orchestra.console:Power off smithi008 2021-07-23T06:20:54.103 INFO:teuthology.kill:2021-07-23 06:20:54,102.102 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2021-07-23T06:20:54.115 INFO:teuthology.kill:2021-07-23 06:20:54,115.115 INFO:teuthology.orchestra.console:Power off for smithi008 completed 2021-07-23T06:20:54.243 INFO:teuthology.kill:2021-07-23 06:20:54,243.243 INFO:teuthology.lock.ops:unlocked smithi187.front.sepia.ceph.com 2021-07-23T06:20:54.255 INFO:teuthology.kill:2021-07-23 06:20:54,255.255 INFO:teuthology.lock.ops:unlocked smithi008.front.sepia.ceph.com 2021-07-23T06:22:54.442 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-07-23T06:22:54.528 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-07-23T06:22:54.529 INFO:teuthology.nuke:Checking targets against current locks 2021-07-23T06:22:54.601 INFO:teuthology.nuke:Not nuking smithi008.front.sepia.ceph.com because description doesn't match 2021-07-23T06:22:54.602 INFO:teuthology.nuke:Not nuking smithi187.front.sepia.ceph.com because description doesn't match