2022-03-30T11:21:29.615 INFO:root:teuthology version: 1.1.0-d877b023
2022-03-30T11:21:29.736 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi040.front.sepia.ceph.com' reimaging
2022-03-30T11:21:29.737 INFO:teuthology.lock.ops:Updating [ubuntu@smithi040.front.sepia.ceph.com]: reset os type and version on server
2022-03-30T11:21:29.738 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server
2022-03-30T11:21:29.756 INFO:teuthology.lock.ops:Node 'ubuntu@smithi040.front.sepia.ceph.com' reimaging is complete
2022-03-30T11:21:29.756 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi065.front.sepia.ceph.com' reimaging
2022-03-30T11:21:29.756 INFO:teuthology.lock.ops:Updating [ubuntu@smithi065.front.sepia.ceph.com]: reset os type and version on server
2022-03-30T11:21:29.757 INFO:teuthology.lock.ops:Updating smithi065.front.sepia.ceph.com on lock server
2022-03-30T11:21:29.775 INFO:teuthology.lock.ops:Node 'ubuntu@smithi065.front.sepia.ceph.com' reimaging is complete
2022-03-30T11:21:29.970 INFO:teuthology.provision.fog.smithi040:Scheduling deploy of centos 8.stream
2022-03-30T11:21:29.980 INFO:teuthology.provision.fog.smithi065:Scheduling deploy of centos 8.stream
2022-03-30T11:21:30.337 INFO:teuthology.orchestra.console:Power off smithi040
2022-03-30T11:21:30.407 INFO:teuthology.orchestra.console:Power off smithi065
2022-03-30T11:21:38.708 INFO:teuthology.orchestra.console:Power off for smithi040 completed
2022-03-30T11:21:38.770 INFO:teuthology.orchestra.console:Power off for smithi065 completed
2022-03-30T11:21:38.810 INFO:teuthology.orchestra.console:Power on smithi040
2022-03-30T11:21:38.871 INFO:teuthology.orchestra.console:Power on smithi065
2022-03-30T11:21:47.176 INFO:teuthology.orchestra.console:Power on for smithi040 completed
2022-03-30T11:21:47.229 INFO:teuthology.orchestra.console:Power on for smithi065 completed
2022-03-30T11:21:47.278 INFO:teuthology.provision.fog.smithi040:Waiting for deploy to finish
2022-03-30T11:21:47.331 INFO:teuthology.provision.fog.smithi065:Waiting for deploy to finish
2022-03-30T11:24:03.666 ERROR:teuthology.orchestra.connection:Error authenticating with smithi040.front.sepia.ceph.com: Authentication failed.
2022-03-30T11:25:35.391 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-30T11:25:55.684 INFO:teuthology.provision.fog.smithi040:Node is ready
2022-03-30T11:25:55.707 INFO:teuthology.orchestra.run.smithi040.stdout:smithi040.front.sepia.ceph.com
2022-03-30T11:25:55.768 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.40 smithi040.front.sepia.ceph.com smithi040
2022-03-30T11:25:56.151 INFO:teuthology.provision.fog.smithi040:Deploy complete!
2022-03-30T11:26:43.555 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-30T11:26:43.802 INFO:teuthology.provision.fog.smithi065:Node is ready
2022-03-30T11:26:43.821 INFO:teuthology.orchestra.run.smithi065.stdout:smithi065.front.sepia.ceph.com
2022-03-30T11:26:43.881 INFO:teuthology.orchestra.run.smithi065.stdout:172.21.15.65 smithi065.front.sepia.ceph.com smithi065
2022-03-30T11:26:44.224 INFO:teuthology.provision.fog.smithi065:Deploy complete!
2022-03-30T11:26:44.375 INFO:teuthology.lock.ops:Checking smithi040.front.sepia.ceph.com
2022-03-30T11:26:44.376 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-30T11:26:44.393 INFO:teuthology.lock.ops:Checking smithi065.front.sepia.ceph.com
2022-03-30T11:26:44.394 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-30T11:26:44.411 INFO:teuthology.lock.ops:Updating [smithi040.front.sepia.ceph.com]: set os type and version on server
2022-03-30T11:26:44.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com'
2022-03-30T11:26:44.663 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64
2022-03-30T11:26:44.721 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream"
2022-03-30T11:26:44.721 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="8"
2022-03-30T11:26:44.722 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos"
2022-03-30T11:26:44.722 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora"
2022-03-30T11:26:44.722 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="8"
2022-03-30T11:26:44.722 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el8"
2022-03-30T11:26:44.723 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 8"
2022-03-30T11:26:44.723 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31"
2022-03-30T11:26:44.723 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-03-30T11:26:44.723 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/"
2022-03-30T11:26:44.724 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-03-30T11:26:44.724 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-03-30T11:26:44.724 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-03-30T11:26:44.725 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server
2022-03-30T11:26:44.744 INFO:teuthology.lock.ops:Updating [smithi065.front.sepia.ceph.com]: set os type and version on server
2022-03-30T11:26:44.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi065.front.sepia.ceph.com'
2022-03-30T11:26:44.938 INFO:teuthology.orchestra.run.smithi065.stdout:x86_64
2022-03-30T11:26:44.996 INFO:teuthology.orchestra.run.smithi065.stdout:NAME="CentOS Stream"
2022-03-30T11:26:44.996 INFO:teuthology.orchestra.run.smithi065.stdout:VERSION="8"
2022-03-30T11:26:44.997 INFO:teuthology.orchestra.run.smithi065.stdout:ID="centos"
2022-03-30T11:26:44.997 INFO:teuthology.orchestra.run.smithi065.stdout:ID_LIKE="rhel fedora"
2022-03-30T11:26:44.997 INFO:teuthology.orchestra.run.smithi065.stdout:VERSION_ID="8"
2022-03-30T11:26:44.997 INFO:teuthology.orchestra.run.smithi065.stdout:PLATFORM_ID="platform:el8"
2022-03-30T11:26:44.998 INFO:teuthology.orchestra.run.smithi065.stdout:PRETTY_NAME="CentOS Stream 8"
2022-03-30T11:26:44.998 INFO:teuthology.orchestra.run.smithi065.stdout:ANSI_COLOR="0;31"
2022-03-30T11:26:44.998 INFO:teuthology.orchestra.run.smithi065.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-03-30T11:26:44.998 INFO:teuthology.orchestra.run.smithi065.stdout:HOME_URL="https://centos.org/"
2022-03-30T11:26:44.998 INFO:teuthology.orchestra.run.smithi065.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-03-30T11:26:44.999 INFO:teuthology.orchestra.run.smithi065.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-03-30T11:26:44.999 INFO:teuthology.orchestra.run.smithi065.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-03-30T11:26:44.999 INFO:teuthology.lock.ops:Updating smithi065.front.sepia.ceph.com on lock server
2022-03-30T11:26:45.099 INFO:teuthology.dispatcher.supervisor:Running job 6768744
2022-03-30T11:26:45.099 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-03-30_07:58:23-fs-wip-vshankar-testing1-20220330-104256-testing-default-smithi/6768744 --name vshankar-2022-03-30_07:58:23-fs-wip-vshankar-testing1-20220330-104256-testing-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/vshankar-2022-03-30_07:58:23-fs-wip-vshankar-testing1-20220330-104256-testing-default-smithi/6768744/orig.config.yaml
2022-03-30T11:26:45.103 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2022-03-30_07:58:23-fs-wip-vshankar-testing1-20220330-104256-testing-default-smithi/6768744
2022-03-30T11:26:45.104 INFO:teuthology.dispatcher.supervisor:Job PID: 6497
2022-03-30T11:26:45.105 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-03-30T17:56:56.036 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-03-30T17:56:56.161 INFO:teuthology.kill:Killing Pids: {6497}
2022-03-30T17:56:57.374 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2022-03-30T17:56:57.375 INFO:teuthology.task.internal:roles: ubuntu@smithi065.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2022-03-30T17:56:57.375 INFO:teuthology.misc:Compressing logs...
2022-03-30T17:56:57.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com'
2022-03-30T17:56:57.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi065.front.sepia.ceph.com'
2022-03-30T17:57:00.907 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/dispatcher/supervisor.py", line 346, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/remote.py", line 509, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/teuthology/orchestra/run.py", line 434, in run
    transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2022-03-30T17:57:00.977 INFO:teuthology.kill:No teuthology processes running
2022-03-30T17:57:02.100 INFO:teuthology.kill:Nuking machines: ['smithi040', 'smithi065']
2022-03-30T17:57:03.015 INFO:teuthology.kill:2022-03-30 17:57:03,015.015 INFO:teuthology.nuke:targets:
2022-03-30T17:57:03.016 INFO:teuthology.kill:  smithi040.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChnpL9nn2Z/Z1/aU0gau50egmABoK2Pe3rgR+ror+e7Yq5Klm17cHVWUIdzlbqfqVqUPZhWpOUp1/8U/AvhzEjnDwstUSfPdHPh+yKbBj/mCeVYtasS6t5Kz5pkmeVaUqQfoSOef+qdL/uBiK7RqM+1trFA/vFSZvYAUlndQtnzwi2hnDn1Jicwf5MCci9Np5pFslG32wRQL6z0lAuWj3iVxiendKnXRjx2whdHM0HBYPAzj0/1Eio0U95OAvAGsm2/7ex3xX/ETyTGjpN9xoCi+OWC8zTKM1VgL69HoYWLkqy7HhKonGrOBm5bNZxxlOpc0A5mbv7LmNkAbtHvEmYr8L7s64ubPxd0e4fn4cT1IbqeHvkC6RBqE8bWngIJt7qqx02KmhxM+qLBdV3YpEdLGy1SSWBdi4bEHBgqzG7sYcgZJ89959ghsGJmikOl+W1KmgVTrXN2sGytjDOq3HYwJzeby4D8kOS206cJ/CjCKd2RKLIm1a8XTXHI5iAjps=
2022-03-30T17:57:03.016 INFO:teuthology.kill:  smithi065.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7ztXCgpXLukYq4Qk7UFkp9Ygo/FwaoSwGpStudtydbmnB/5BzdbDijTNDgpusF7u6UqrDCOpbCCC3Q83Lxpc1RFCGihRSmba35tZtpPx/ekIP3+My+/XPoXxjbRdd3sDT3zi8RSh63v7IK2Q+JaOlei7UcAe0Kzl4nsWeq0iayacoXuHmoiWcSrhJIfFR1TKkiPJqMs/iviMgW5zTgAEva4I+vMfhiLLgx6iMH+aXU9JC2xd0omHqleFuhEF3x8JbvXioO91sBoum8/C+mQVG1fckdggMzbL4RsPMQqkvF/2o+7DVlcKxxkh+MjTGMQyM+arw8fDAgigQrqLT+B84SxN1TgPG8yiWs3I7Nauw3Tl9kFM5P6XALdl2ePBGyqGiEMjL5AMaf5KUagMBd4GhhVkbBJytPYYS4EqU1+L2QJ0WokAlSo9thoK/R7PZJn3fHRQFyGMRI4S3y/JapWgoMQE5j8ig4aO63MyCoy+rDu2ohlYu1rCH1M+FOZFL480=
2022-03-30T17:57:03.044 INFO:teuthology.kill:2022-03-30 17:57:03,043.043 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-30T17:57:03.047 INFO:teuthology.kill:2022-03-30 17:57:03,047.047 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-30T17:57:03.124 INFO:teuthology.kill:2022-03-30 17:57:03,124.124 INFO:teuthology.orchestra.console:Power off smithi065
2022-03-30T17:57:03.137 INFO:teuthology.kill:2022-03-30 17:57:03,137.137 INFO:teuthology.orchestra.console:Power off smithi040
2022-03-30T17:57:11.491 INFO:teuthology.kill:2022-03-30 17:57:11,491.491 INFO:teuthology.orchestra.console:Power off for smithi065 completed
2022-03-30T17:57:11.497 INFO:teuthology.kill:2022-03-30 17:57:11,496.496 INFO:teuthology.orchestra.console:Power off for smithi040 completed
2022-03-30T17:57:11.639 INFO:teuthology.kill:2022-03-30 17:57:11,638.638 INFO:teuthology.lock.ops:unlocked: smithi065.front.sepia.ceph.com
2022-03-30T17:57:11.643 INFO:teuthology.kill:2022-03-30 17:57:11,643.643 INFO:teuthology.lock.ops:unlocked: smithi040.front.sepia.ceph.com
2022-03-30T17:59:11.868 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-03-30T17:59:12.001 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi040 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-03-21_02:01:03-rbd-quincy-distro-default-smithi/6749426
2022-03-30T17:59:12.001 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi065 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-03-21_02:01:03-rbd-quincy-distro-default-smithi/6749426