2022-04-21T13:01:40.877 INFO:root:teuthology version: 1.1.0-b37d6865 2022-04-21T13:01:41.028 INFO:teuthology.lock.ops:Start node 'smithi039.front.sepia.ceph.com' reimaging 2022-04-21T13:01:41.029 INFO:teuthology.lock.ops:Updating [smithi039.front.sepia.ceph.com]: reset os type and version on server 2022-04-21T13:01:41.029 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2022-04-21T13:01:41.048 INFO:teuthology.lock.ops:Node 'smithi039.front.sepia.ceph.com' reimaging is complete 2022-04-21T13:01:41.049 INFO:teuthology.lock.ops:Start node 'smithi047.front.sepia.ceph.com' reimaging 2022-04-21T13:01:41.049 INFO:teuthology.lock.ops:Updating [smithi047.front.sepia.ceph.com]: reset os type and version on server 2022-04-21T13:01:41.049 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2022-04-21T13:01:41.068 INFO:teuthology.lock.ops:Node 'smithi047.front.sepia.ceph.com' reimaging is complete 2022-04-21T13:01:41.258 INFO:teuthology.provision.fog.smithi039:Scheduling deploy of centos 8.stream 2022-04-21T13:01:41.279 INFO:teuthology.provision.fog.smithi047:Scheduling deploy of centos 8.stream 2022-04-21T13:01:41.647 INFO:teuthology.orchestra.console:Power off smithi039 2022-04-21T13:01:41.789 INFO:teuthology.orchestra.console:Power off for smithi039 completed 2022-04-21T13:01:41.800 INFO:teuthology.orchestra.console:Power off smithi047 2022-04-21T13:01:41.891 INFO:teuthology.orchestra.console:Power on smithi039 2022-04-21T13:01:41.949 INFO:teuthology.orchestra.console:Power off for smithi047 completed 2022-04-21T13:01:42.050 INFO:teuthology.orchestra.console:Power on smithi047 2022-04-21T13:01:50.261 INFO:teuthology.orchestra.console:Power on for smithi039 completed 2022-04-21T13:01:50.362 INFO:teuthology.provision.fog.smithi039:Waiting for deploy to finish 2022-04-21T13:01:50.425 INFO:teuthology.orchestra.console:Power on for smithi047 completed 2022-04-21T13:01:50.526 INFO:teuthology.provision.fog.smithi047:Waiting for deploy to finish 2022-04-21T13:05:23.672 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-04-21T13:05:49.378 INFO:teuthology.provision.fog.smithi047:Node is ready 2022-04-21T13:05:49.398 INFO:teuthology.orchestra.run.smithi047.stdout:smithi047.front.sepia.ceph.com 2022-04-21T13:05:49.455 INFO:teuthology.orchestra.run.smithi047.stdout:172.21.15.47 smithi047.front.sepia.ceph.com smithi047 2022-04-21T13:05:49.839 INFO:teuthology.provision.fog.smithi047:Deploy complete! 2022-04-21T13:07:10.926 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-04-21T13:07:11.248 INFO:teuthology.provision.fog.smithi039:Node is ready 2022-04-21T13:07:11.269 INFO:teuthology.orchestra.run.smithi039.stdout:smithi039.front.sepia.ceph.com 2022-04-21T13:07:11.335 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39 smithi039.front.sepia.ceph.com smithi039 2022-04-21T13:07:11.687 INFO:teuthology.provision.fog.smithi039:Deploy complete! 2022-04-21T13:07:11.883 INFO:teuthology.lock.ops:Checking smithi039.front.sepia.ceph.com 2022-04-21T13:07:11.883 INFO:teuthology.lock.ops:New key found. Updating... 2022-04-21T13:07:11.910 INFO:teuthology.lock.ops:Checking smithi047.front.sepia.ceph.com 2022-04-21T13:07:11.910 INFO:teuthology.lock.ops:New key found. Updating... 2022-04-21T13:07:11.928 INFO:teuthology.lock.ops:Updating [smithi039.front.sepia.ceph.com]: set os type and version on server 2022-04-21T13:07:11.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2022-04-21T13:07:12.122 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2022-04-21T13:07:12.179 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="CentOS Stream" 2022-04-21T13:07:12.204 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="8" 2022-04-21T13:07:12.205 INFO:teuthology.orchestra.run.smithi039.stdout:ID="centos" 2022-04-21T13:07:12.205 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="rhel fedora" 2022-04-21T13:07:12.205 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="8" 2022-04-21T13:07:12.205 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el8" 2022-04-21T13:07:12.205 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="CentOS Stream 8" 2022-04-21T13:07:12.205 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31" 2022-04-21T13:07:12.205 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-04-21T13:07:12.205 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://centos.org/" 2022-04-21T13:07:12.206 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-04-21T13:07:12.206 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-04-21T13:07:12.207 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-04-21T13:07:12.208 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2022-04-21T13:07:12.229 INFO:teuthology.lock.ops:Updating [smithi047.front.sepia.ceph.com]: set os type and version on server 2022-04-21T13:07:12.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2022-04-21T13:07:12.535 INFO:teuthology.orchestra.run.smithi047.stdout:x86_64 2022-04-21T13:07:12.598 INFO:teuthology.orchestra.run.smithi047.stdout:NAME="CentOS Stream" 2022-04-21T13:07:12.598 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION="8" 2022-04-21T13:07:12.599 INFO:teuthology.orchestra.run.smithi047.stdout:ID="centos" 2022-04-21T13:07:12.599 INFO:teuthology.orchestra.run.smithi047.stdout:ID_LIKE="rhel fedora" 2022-04-21T13:07:12.599 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_ID="8" 2022-04-21T13:07:12.599 INFO:teuthology.orchestra.run.smithi047.stdout:PLATFORM_ID="platform:el8" 2022-04-21T13:07:12.600 INFO:teuthology.orchestra.run.smithi047.stdout:PRETTY_NAME="CentOS Stream 8" 2022-04-21T13:07:12.601 INFO:teuthology.orchestra.run.smithi047.stdout:ANSI_COLOR="0;31" 2022-04-21T13:07:12.601 INFO:teuthology.orchestra.run.smithi047.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-04-21T13:07:12.601 INFO:teuthology.orchestra.run.smithi047.stdout:HOME_URL="https://centos.org/" 2022-04-21T13:07:12.601 INFO:teuthology.orchestra.run.smithi047.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-04-21T13:07:12.602 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-04-21T13:07:12.602 INFO:teuthology.orchestra.run.smithi047.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-04-21T13:07:12.603 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2022-04-21T13:07:12.715 INFO:teuthology.dispatcher.supervisor:Running job 6798791 2022-04-21T13:07:12.715 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b37d686568ea3c673197ca892d3fb267463611f8/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-04-21_04:42:50-fs-wip-vshankar-testing-20220420-113951-testing-default-smithi/6798791 --name vshankar-2022-04-21_04:42:50-fs-wip-vshankar-testing-20220420-113951-testing-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/vshankar-2022-04-21_04:42:50-fs-wip-vshankar-testing-20220420-113951-testing-default-smithi/6798791/orig.config.yaml 2022-04-21T13:07:12.720 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2022-04-21_04:42:50-fs-wip-vshankar-testing-20220420-113951-testing-default-smithi/6798791 2022-04-21T13:07:12.720 INFO:teuthology.dispatcher.supervisor:Job PID: 27455 2022-04-21T13:07:12.721 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-04-21T19:37:39.020 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-04-21T19:37:39.219 INFO:teuthology.kill:Killing Pids: {27455} 2022-04-21T19:37:40.295 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2022-04-21T19:37:40.296 INFO:teuthology.task.internal:roles: ubuntu@smithi047.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2022-04-21T19:37:40.296 INFO:teuthology.misc:Compressing logs... 2022-04-21T19:37:40.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2022-04-21T19:37:40.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2022-04-21T19:37:42.315 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b37d686568ea3c673197ca892d3fb267463611f8/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b37d686568ea3c673197ca892d3fb267463611f8/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b37d686568ea3c673197ca892d3fb267463611f8/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b37d686568ea3c673197ca892d3fb267463611f8/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b37d686568ea3c673197ca892d3fb267463611f8/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b37d686568ea3c673197ca892d3fb267463611f8/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b37d686568ea3c673197ca892d3fb267463611f8/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2022-04-21T19:37:42.380 INFO:teuthology.kill:No teuthology processes running 2022-04-21T19:37:43.395 INFO:teuthology.kill:Nuking machines: ['smithi039', 'smithi047'] 2022-04-21T19:37:44.243 INFO:teuthology.kill:2022-04-21 19:37:44,242.242 INFO:teuthology.nuke:targets: 2022-04-21T19:37:44.243 INFO:teuthology.kill: smithi039.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDte0IzjwGDptfK/hryltDJaBuGZCtdCINqW98JhxhnTmXW8hogiqOcZUi92ILF9i+C7uEafGiz2cH1ajzoP9W9HK5h2DCpygzAPwcxdNDeXtsIVnPyr02Pt860b/0keMJVvo1FD9vzjlktGnqpRwTH9dlm8adJz+6NFhLA3phm/iDWLwhhQUKcd6GACF3dFZYbV7jUsi5ZoC1wyBTnFukwO/fqza2Qr1296RFJ0LiGzkHIWLIYPPWphBUCYgEWvjCwANus3AdUdU2B4/HblRqsQEdkgg5CmyiiCPitOIBjQ8qAGdyfy5N2kJeqz3zSOuwQgK7rMHAYtJpKi8Nj2BF9axb+8HVWBapvzKyIQQ3uVVXRXShGad0UWjIeryDw2ScCJzkFmZiRZcjG1OJY7I4HhdtF8KGp0XpIKi8pSaNxXkKxDC8Gm8oUdOx/ghEj5ZqouwPdoSLsGo/okyfrlPTpPP0NwX0gFBekwQ46/oE0bahp04XELXgdaufKXipFXs= 2022-04-21T19:37:44.243 INFO:teuthology.kill: smithi047.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4zJOb3kSUstmn64m0t/oyt0RRG5O5L+3ZTKi8iFP0Fimp2/8obVIcTIa4yYKjXGoWHoz5Mto2o18fb0ZdAkUQHfIGOzd59sgwGmKwHFfqF5xd650V1NkKqjwpYEogvQO9DGvbL4gWJuR+3CqJUAtMVfD0X5fkzfLbhVi2F2MT/U7jiQL33JcCdqKqCzOIN0ZLp/QQ3yglYme3De8flDYCanur9CZcdE10DohIfse8Ub4oMf52l4VOQO0GQWyvwyB1zDG+I3BHqnSFpmB4qKMFF12VEyk2/zzmJ89kxQsMzlm2yYjqMqSORRj5tM7Q6L3gD/hhJ+47wNYfmuZxG88eYxJhLSOHIfXh4quUhKZzgxlkgOjAUMJGUUifJp5LkYXl5FFLcssfqxrh5wPiy1Ig7ysdJ0oUxin0jF8GeqU9TTDsqrAGOTyo0IOuXEIUBSir4LECpzGNxp3wCkKxt2Bs+H9y+Vv6fZJ4gVW88S3ruGSQjSxNMhtxm8tsYK7FGy8= 2022-04-21T19:37:44.270 INFO:teuthology.kill:2022-04-21 19:37:44,270.270 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-04-21T19:37:44.273 INFO:teuthology.kill:2022-04-21 19:37:44,273.273 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-04-21T19:37:44.348 INFO:teuthology.kill:2022-04-21 19:37:44,347.347 INFO:teuthology.orchestra.console:Power off smithi039 2022-04-21T19:37:44.361 INFO:teuthology.kill:2022-04-21 19:37:44,361.361 INFO:teuthology.orchestra.console:Power off smithi047 2022-04-21T19:37:52.724 INFO:teuthology.kill:2022-04-21 19:37:52,724.724 INFO:teuthology.orchestra.console:Power off for smithi047 completed 2022-04-21T19:37:52.869 INFO:teuthology.kill:2022-04-21 19:37:52,868.868 INFO:teuthology.lock.ops:unlocked: smithi047.front.sepia.ceph.com 2022-04-21T19:37:56.831 INFO:teuthology.kill:2022-04-21 19:37:56,830.830 INFO:teuthology.orchestra.console:Power off for smithi039 completed 2022-04-21T19:37:56.981 INFO:teuthology.kill:2022-04-21 19:37:56,981.981 INFO:teuthology.lock.ops:unlocked: smithi039.front.sepia.ceph.com 2022-04-21T19:39:57.227 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15