2022-04-06T13:15:41.085 INFO:root:teuthology version: 1.1.0-3dd79763 2022-04-06T13:15:41.202 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi070.front.sepia.ceph.com' reimaging 2022-04-06T13:15:41.203 INFO:teuthology.lock.ops:Updating [ubuntu@smithi070.front.sepia.ceph.com]: reset os type and version on server 2022-04-06T13:15:41.204 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2022-04-06T13:15:41.221 INFO:teuthology.lock.ops:Node 'ubuntu@smithi070.front.sepia.ceph.com' reimaging is complete 2022-04-06T13:15:41.221 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi074.front.sepia.ceph.com' reimaging 2022-04-06T13:15:41.221 INFO:teuthology.lock.ops:Updating [ubuntu@smithi074.front.sepia.ceph.com]: reset os type and version on server 2022-04-06T13:15:41.222 INFO:teuthology.lock.ops:Updating smithi074.front.sepia.ceph.com on lock server 2022-04-06T13:15:41.240 INFO:teuthology.lock.ops:Node 'ubuntu@smithi074.front.sepia.ceph.com' reimaging is complete 2022-04-06T13:15:41.426 INFO:teuthology.provision.fog.smithi070:Scheduling deploy of centos 8.stream 2022-04-06T13:15:41.444 INFO:teuthology.provision.fog.smithi074:Scheduling deploy of centos 8.stream 2022-04-06T13:15:41.720 INFO:teuthology.orchestra.console:Power off smithi070 2022-04-06T13:15:41.759 INFO:teuthology.orchestra.console:Power off smithi074 2022-04-06T13:15:50.093 INFO:teuthology.orchestra.console:Power off for smithi070 completed 2022-04-06T13:15:50.146 INFO:teuthology.orchestra.console:Power off for smithi074 completed 2022-04-06T13:15:50.195 INFO:teuthology.orchestra.console:Power on smithi070 2022-04-06T13:15:50.247 INFO:teuthology.orchestra.console:Power on smithi074 2022-04-06T13:15:58.562 INFO:teuthology.orchestra.console:Power on for smithi070 completed 2022-04-06T13:15:58.609 INFO:teuthology.orchestra.console:Power on for smithi074 completed 2022-04-06T13:15:58.664 INFO:teuthology.provision.fog.smithi070:Waiting for deploy to finish 2022-04-06T13:15:58.711 INFO:teuthology.provision.fog.smithi074:Waiting for deploy to finish 2022-04-06T13:20:08.451 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-04-06T13:20:34.035 INFO:teuthology.provision.fog.smithi074:Node is ready 2022-04-06T13:20:34.054 INFO:teuthology.orchestra.run.smithi074.stdout:smithi074.front.sepia.ceph.com 2022-04-06T13:20:34.110 INFO:teuthology.orchestra.run.smithi074.stdout:172.21.15.74 smithi074.front.sepia.ceph.com smithi074 2022-04-06T13:20:34.510 INFO:teuthology.provision.fog.smithi074:Deploy complete! 2022-04-06T13:20:36.794 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-04-06T13:20:37.092 INFO:teuthology.provision.fog.smithi070:Node is ready 2022-04-06T13:20:37.111 INFO:teuthology.orchestra.run.smithi070.stdout:smithi070.front.sepia.ceph.com 2022-04-06T13:20:37.169 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70 smithi070.front.sepia.ceph.com smithi070 2022-04-06T13:20:37.513 INFO:teuthology.provision.fog.smithi070:Deploy complete! 2022-04-06T13:20:37.688 INFO:teuthology.lock.ops:Checking smithi070.front.sepia.ceph.com 2022-04-06T13:20:37.689 INFO:teuthology.lock.ops:New key found. Updating... 2022-04-06T13:20:37.707 INFO:teuthology.lock.ops:Checking smithi074.front.sepia.ceph.com 2022-04-06T13:20:37.707 INFO:teuthology.lock.ops:New key found. Updating... 2022-04-06T13:20:37.728 INFO:teuthology.lock.ops:Updating [smithi070.front.sepia.ceph.com]: set os type and version on server 2022-04-06T13:20:37.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2022-04-06T13:20:37.920 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2022-04-06T13:20:37.977 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="CentOS Stream" 2022-04-06T13:20:37.978 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="8" 2022-04-06T13:20:37.978 INFO:teuthology.orchestra.run.smithi070.stdout:ID="centos" 2022-04-06T13:20:37.978 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="rhel fedora" 2022-04-06T13:20:37.979 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="8" 2022-04-06T13:20:37.979 INFO:teuthology.orchestra.run.smithi070.stdout:PLATFORM_ID="platform:el8" 2022-04-06T13:20:37.979 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="CentOS Stream 8" 2022-04-06T13:20:37.979 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31" 2022-04-06T13:20:37.980 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-04-06T13:20:37.980 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://centos.org/" 2022-04-06T13:20:37.980 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-04-06T13:20:37.980 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-04-06T13:20:37.980 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-04-06T13:20:37.982 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2022-04-06T13:20:38.000 INFO:teuthology.lock.ops:Updating [smithi074.front.sepia.ceph.com]: set os type and version on server 2022-04-06T13:20:38.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi074.front.sepia.ceph.com' 2022-04-06T13:20:38.202 INFO:teuthology.orchestra.run.smithi074.stdout:x86_64 2022-04-06T13:20:38.260 INFO:teuthology.orchestra.run.smithi074.stdout:NAME="CentOS Stream" 2022-04-06T13:20:38.260 INFO:teuthology.orchestra.run.smithi074.stdout:VERSION="8" 2022-04-06T13:20:38.261 INFO:teuthology.orchestra.run.smithi074.stdout:ID="centos" 2022-04-06T13:20:38.261 INFO:teuthology.orchestra.run.smithi074.stdout:ID_LIKE="rhel fedora" 2022-04-06T13:20:38.261 INFO:teuthology.orchestra.run.smithi074.stdout:VERSION_ID="8" 2022-04-06T13:20:38.261 INFO:teuthology.orchestra.run.smithi074.stdout:PLATFORM_ID="platform:el8" 2022-04-06T13:20:38.262 INFO:teuthology.orchestra.run.smithi074.stdout:PRETTY_NAME="CentOS Stream 8" 2022-04-06T13:20:38.262 INFO:teuthology.orchestra.run.smithi074.stdout:ANSI_COLOR="0;31" 2022-04-06T13:20:38.262 INFO:teuthology.orchestra.run.smithi074.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-04-06T13:20:38.262 INFO:teuthology.orchestra.run.smithi074.stdout:HOME_URL="https://centos.org/" 2022-04-06T13:20:38.262 INFO:teuthology.orchestra.run.smithi074.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-04-06T13:20:38.263 INFO:teuthology.orchestra.run.smithi074.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-04-06T13:20:38.263 INFO:teuthology.orchestra.run.smithi074.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-04-06T13:20:38.264 INFO:teuthology.lock.ops:Updating smithi074.front.sepia.ceph.com on lock server 2022-04-06T13:20:38.360 INFO:teuthology.dispatcher.supervisor:Running job 6779112 2022-04-06T13:20:38.361 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_3dd797633f37e2d7e2c4ae3ef9ac35a188c2028d/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-04-06_12:57:19-fs-wip-vshankar-testing-20220403-170344-testing-default-smithi/6779112 --name vshankar-2022-04-06_12:57:19-fs-wip-vshankar-testing-20220403-170344-testing-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/vshankar-2022-04-06_12:57:19-fs-wip-vshankar-testing-20220403-170344-testing-default-smithi/6779112/orig.config.yaml 2022-04-06T13:20:38.364 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2022-04-06_12:57:19-fs-wip-vshankar-testing-20220403-170344-testing-default-smithi/6779112 2022-04-06T13:20:38.365 INFO:teuthology.dispatcher.supervisor:Job PID: 12576 2022-04-06T13:20:38.366 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-04-06T19:51:07.295 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-04-06T19:51:07.486 INFO:teuthology.kill:Killing Pids: {12576} 2022-04-06T19:51:08.654 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2022-04-06T19:51:08.655 INFO:teuthology.task.internal:roles: ubuntu@smithi074.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2022-04-06T19:51:08.656 INFO:teuthology.misc:Compressing logs... 2022-04-06T19:51:08.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2022-04-06T19:51:08.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi074.front.sepia.ceph.com' 2022-04-06T19:51:10.571 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_3dd797633f37e2d7e2c4ae3ef9ac35a188c2028d/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3dd797633f37e2d7e2c4ae3ef9ac35a188c2028d/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3dd797633f37e2d7e2c4ae3ef9ac35a188c2028d/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_3dd797633f37e2d7e2c4ae3ef9ac35a188c2028d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_3dd797633f37e2d7e2c4ae3ef9ac35a188c2028d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_3dd797633f37e2d7e2c4ae3ef9ac35a188c2028d/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3dd797633f37e2d7e2c4ae3ef9ac35a188c2028d/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2022-04-06T19:51:10.644 INFO:teuthology.kill:No teuthology processes running 2022-04-06T19:51:11.812 INFO:teuthology.kill:Nuking machines: ['smithi070', 'smithi074'] 2022-04-06T19:51:12.793 INFO:teuthology.kill:2022-04-06 19:51:12,792.792 INFO:teuthology.nuke:targets: 2022-04-06T19:51:12.794 INFO:teuthology.kill: smithi070.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbOPp1WX6W+W2T16MgflbXX6OPzh9WZRsaUJLqwWt1MCnZQ05EITXgWnZ/6YzcBucnHKHbw7Uus6ZfDwWDvhJuGlvWx0GVj8aSyIWcK4/6HdBniQfLYydg1RwpUCrvpV+QjlhqyGQU/mQos4HHePBADNZHqrVCcWZhd92CVbNg2H41URgXnm5whJajG5OYMG8UEIZMcQPEEwatxVhUVysY8ki28qB4bTemxClA5jsL3f/cH0T3rCQuyI8if60THdtZCG3Xkn6HFCTkKu3HxBDlVUDjpzn7UUqdt+Qy2RBOPPos0mOsitP7uqfBqDVNmP3NdiIusdrvZileWE8EAS/bRfl/tkH76fT3jWwFAZeKnvq0hmadWmclhBlmrW2FX4zd1jwHNNSBHg8VI+eH4+YCiISnt/IjJTqSckDKqiwFKc+ZdahI1/DdLH1lWf6QefMkSPmzaCqmcB2IurR96ptSrVzLkS1DjsJlOar7q+txZGCZ98MwAZWbsWKzw23xeLc= 2022-04-06T19:51:12.794 INFO:teuthology.kill: smithi074.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/dztpPCEMfZQSp7yn8Ry0f65pyQejto0D8wPKv9rFJp0fWDXwCu998tJVG84n045nYE9soYr6WNi+VwHl3/gOuv12L1ju60l6gomx+S6GwZgjDckQuKP6kiVErI8vryIQx7gKZMKpf7pdHBeyQqwiGzxaljIl4vQIGUTkp3fe/LeS3Ts8+XKkocebBdDfGhdadu9aQdp5NLubMytdv0rE6LM/3tEjnU76wNv2IhJuOgGN9vSCTJInSxe4WVKMgZrnMEUk+Z6AF41DQdU1xKqKXad6kzLv3zDUkVzZvfqDbm2zIYOKRU1M6NFdTGzOMElr6hFuZNyMxFqDhobN/MXaKXqM6osh5UgZfRZKNTQk9QqD4Bzl0B9rh3zoHrE8NhQMxZ7GEk6mk/O/Q6kpSVi/6GBLxNiOcOrhbEjm7f5B0g+PU1glJalIP4nM/YjeiYUg1aZmyCSSaRHGNH3zxtufI8whZvz10LwspzKV0rO2btn92ecm+YOzp5a6I1iNyvk= 2022-04-06T19:51:12.819 INFO:teuthology.kill:2022-04-06 19:51:12,819.819 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-04-06T19:51:12.822 INFO:teuthology.kill:2022-04-06 19:51:12,822.822 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-04-06T19:51:12.891 INFO:teuthology.kill:2022-04-06 19:51:12,891.891 INFO:teuthology.orchestra.console:Power off smithi070 2022-04-06T19:51:12.907 INFO:teuthology.kill:2022-04-06 19:51:12,906.906 INFO:teuthology.orchestra.console:Power off smithi074 2022-04-06T19:51:21.270 INFO:teuthology.kill:2022-04-06 19:51:21,269.269 INFO:teuthology.orchestra.console:Power off for smithi074 completed 2022-04-06T19:51:21.416 INFO:teuthology.kill:2022-04-06 19:51:21,415.415 INFO:teuthology.lock.ops:unlocked: smithi074.front.sepia.ceph.com 2022-04-06T19:51:25.372 INFO:teuthology.kill:2022-04-06 19:51:25,372.372 INFO:teuthology.orchestra.console:Power off for smithi070 completed 2022-04-06T19:51:25.519 INFO:teuthology.kill:2022-04-06 19:51:25,519.519 INFO:teuthology.lock.ops:unlocked: smithi070.front.sepia.ceph.com 2022-04-06T19:53:25.835 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-04-06T19:53:25.950 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi070 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-04-06_16:35:43-rados-wip-yuri5-testing-2022-04-05-1720-distro-default-smithi/6779886 2022-04-06T19:53:25.950 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi074 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-04-06_16:35:43-rados-wip-yuri5-testing-2022-04-05-1720-distro-default-smithi/6779886