2023-04-06T20:27:19.495 INFO:root:teuthology version: 0.0.1.dev93+gd7c4062
2023-04-06T20:27:19.635 INFO:teuthology.lock.ops:Start node 'smithi027.front.sepia.ceph.com' reimaging
2023-04-06T20:27:19.636 INFO:teuthology.lock.ops:Updating [smithi027.front.sepia.ceph.com]: reset os type and version on server
2023-04-06T20:27:19.636 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server
2023-04-06T20:27:19.667 INFO:teuthology.lock.ops:Node 'smithi027.front.sepia.ceph.com' reimaging is complete
2023-04-06T20:27:19.667 INFO:teuthology.lock.ops:Start node 'smithi063.front.sepia.ceph.com' reimaging
2023-04-06T20:27:19.668 INFO:teuthology.lock.ops:Updating [smithi063.front.sepia.ceph.com]: reset os type and version on server
2023-04-06T20:27:19.668 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server
2023-04-06T20:27:19.692 INFO:teuthology.lock.ops:Node 'smithi063.front.sepia.ceph.com' reimaging is complete
2023-04-06T20:27:19.829 INFO:teuthology.provision.fog.smithi027:Scheduling deploy of centos 8.stream
2023-04-06T20:27:19.845 INFO:teuthology.provision.fog.smithi063:Scheduling deploy of centos 8.stream
2023-04-06T20:27:20.079 INFO:teuthology.orchestra.console:Power off smithi027
2023-04-06T20:27:20.093 INFO:teuthology.orchestra.console:Power off smithi063
2023-04-06T20:27:28.454 INFO:teuthology.orchestra.console:Power off for smithi027 completed
2023-04-06T20:27:28.555 INFO:teuthology.orchestra.console:Power on smithi027
2023-04-06T20:27:32.573 INFO:teuthology.orchestra.console:Power off for smithi063 completed
2023-04-06T20:27:32.675 INFO:teuthology.orchestra.console:Power on smithi063
2023-04-06T20:27:36.919 INFO:teuthology.orchestra.console:Power on for smithi027 completed
2023-04-06T20:27:37.020 INFO:teuthology.provision.fog.smithi027:Waiting for deploy to finish
2023-04-06T20:27:41.040 INFO:teuthology.orchestra.console:Power on for smithi063 completed
2023-04-06T20:27:41.141 INFO:teuthology.provision.fog.smithi063:Waiting for deploy to finish
2023-04-06T20:30:29.993 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.63
2023-04-06T20:30:39.077 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.63
2023-04-06T20:30:48.133 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.63
2023-04-06T20:30:57.189 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.63
2023-04-06T20:31:06.245 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.63
2023-04-06T20:31:08.131 ERROR:teuthology.orchestra.connection:Error authenticating with smithi027.front.sepia.ceph.com: Authentication failed.
2023-04-06T20:31:15.301 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.63
2023-04-06T20:31:24.357 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.63
2023-04-06T20:32:02.258 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-04-06T20:32:09.135 WARNING:teuthology.provision.fog:timed out
2023-04-06T20:32:22.813 INFO:teuthology.provision.fog.smithi063:Node is ready
2023-04-06T20:32:22.835 INFO:teuthology.orchestra.run.smithi063.stdout:smithi001
2023-04-06T20:32:22.894 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.1 smithi001 smithi001
2023-04-06T20:32:23.271 INFO:teuthology.provision.fog.smithi063:Deploy complete!
2023-04-06T20:32:33.541 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.27
2023-04-06T20:32:42.597 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.27
2023-04-06T20:32:51.653 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.27
2023-04-06T20:33:57.657 WARNING:teuthology.provision.fog:timed out
2023-04-06T20:34:04.018 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-04-06T20:34:04.290 INFO:teuthology.provision.fog.smithi027:Node is ready
2023-04-06T20:34:04.309 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027.front.sepia.ceph.com
2023-04-06T20:34:04.367 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27 smithi027.front.sepia.ceph.com smithi027
2023-04-06T20:34:04.740 INFO:teuthology.provision.fog.smithi027:Deploy complete!
2023-04-06T20:34:04.931 INFO:teuthology.lock.ops:Checking smithi027.front.sepia.ceph.com
2023-04-06T20:34:04.932 INFO:teuthology.lock.ops:New key found. Updating...
2023-04-06T20:34:04.952 INFO:teuthology.lock.ops:Checking smithi063.front.sepia.ceph.com
2023-04-06T20:34:04.952 INFO:teuthology.lock.ops:New key found. Updating...
2023-04-06T20:34:04.978 INFO:teuthology.lock.ops:Updating [smithi027.front.sepia.ceph.com]: set os type and version on server
2023-04-06T20:34:04.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-04-06T20:34:05.167 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64
2023-04-06T20:34:05.225 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream"
2023-04-06T20:34:05.225 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="8"
2023-04-06T20:34:05.225 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos"
2023-04-06T20:34:05.225 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora"
2023-04-06T20:34:05.226 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="8"
2023-04-06T20:34:05.226 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el8"
2023-04-06T20:34:05.226 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 8"
2023-04-06T20:34:05.226 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31"
2023-04-06T20:34:05.226 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-04-06T20:34:05.226 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/"
2023-04-06T20:34:05.227 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-04-06T20:34:05.227 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-04-06T20:34:05.227 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-04-06T20:34:05.228 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server
2023-04-06T20:34:05.252 INFO:teuthology.lock.ops:Updating [smithi063.front.sepia.ceph.com]: set os type and version on server
2023-04-06T20:34:05.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com'
2023-04-06T20:34:05.518 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64
2023-04-06T20:34:05.577 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="CentOS Stream"
2023-04-06T20:34:05.577 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="8"
2023-04-06T20:34:05.577 INFO:teuthology.orchestra.run.smithi063.stdout:ID="centos"
2023-04-06T20:34:05.578 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE="rhel fedora"
2023-04-06T20:34:05.578 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="8"
2023-04-06T20:34:05.578 INFO:teuthology.orchestra.run.smithi063.stdout:PLATFORM_ID="platform:el8"
2023-04-06T20:34:05.578 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="CentOS Stream 8"
2023-04-06T20:34:05.578 INFO:teuthology.orchestra.run.smithi063.stdout:ANSI_COLOR="0;31"
2023-04-06T20:34:05.578 INFO:teuthology.orchestra.run.smithi063.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-04-06T20:34:05.578 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://centos.org/"
2023-04-06T20:34:05.579 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-04-06T20:34:05.579 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-04-06T20:34:05.579 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-04-06T20:34:05.579 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server
2023-04-06T20:34:05.694 INFO:teuthology.dispatcher.supervisor:Running job 7232551
2023-04-06T20:34:05.694 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_d7c4062c8e48ef4dca373feb7285bdfb014e59b7/virtualenv/bin/teuthology -v --owner scheduled_adking@teuthology --archive /home/teuthworker/archive/adking-2023-04-05_05:00:01-orch:cephadm-wip-adk-testing-2023-04-04-2123-distro-default-smithi/7232551 --name adking-2023-04-05_05:00:01-orch:cephadm-wip-adk-testing-2023-04-04-2123-distro-default-smithi --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/adking-2023-04-05_05:00:01-orch:cephadm-wip-adk-testing-2023-04-04-2123-distro-default-smithi/7232551/orig.config.yaml
2023-04-06T20:34:05.698 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/adking-2023-04-05_05:00:01-orch:cephadm-wip-adk-testing-2023-04-04-2123-distro-default-smithi/7232551
2023-04-06T20:34:05.699 INFO:teuthology.dispatcher.supervisor:Job PID: 26725
2023-04-06T20:34:05.699 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-04-07T08:34:28.783 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-04-07T08:34:29.222 INFO:teuthology.kill:Killing Pids: {26725}
2023-04-07T08:34:38.396 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2023-04-07T08:34:38.396 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2023-04-07T08:34:38.397 INFO:teuthology.misc:Compressing logs...
2023-04-07T08:34:38.397 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-04-07T08:34:38.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com'
2023-04-07T08:34:53.698 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner
2023-04-07T08:34:53.726 ERROR:paramiko.transport:Traceback (most recent call last):
2023-04-07T08:34:53.726 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_d7c4062c8e48ef4dca373feb7285bdfb014e59b7/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-04-07T08:34:53.726 ERROR:paramiko.transport:    buf = self.packetizer.readline(timeout)
2023-04-07T08:34:53.727 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_d7c4062c8e48ef4dca373feb7285bdfb014e59b7/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline
2023-04-07T08:34:53.727 ERROR:paramiko.transport:    buf += self._read_timeout(timeout)
2023-04-07T08:34:53.727 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_d7c4062c8e48ef4dca373feb7285bdfb014e59b7/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 622, in _read_timeout
2023-04-07T08:34:53.727 ERROR:paramiko.transport:    raise socket.timeout()
2023-04-07T08:34:53.728 ERROR:paramiko.transport:socket.timeout
2023-04-07T08:34:53.728 ERROR:paramiko.transport:
2023-04-07T08:34:53.728 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2023-04-07T08:34:53.728 ERROR:paramiko.transport:
2023-04-07T08:34:53.728 ERROR:paramiko.transport:Traceback (most recent call last):
2023-04-07T08:34:53.729 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_d7c4062c8e48ef4dca373feb7285bdfb014e59b7/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2093, in run
2023-04-07T08:34:53.729 ERROR:paramiko.transport:    self._check_banner()
2023-04-07T08:34:53.729 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_d7c4062c8e48ef4dca373feb7285bdfb014e59b7/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2274, in _check_banner
2023-04-07T08:34:53.729 ERROR:paramiko.transport:    raise SSHException(
2023-04-07T08:34:53.730 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-04-07T08:34:53.730 ERROR:paramiko.transport:
2023-04-07T08:34:53.730 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_d7c4062c8e48ef4dca373feb7285bdfb014e59b7/teuthology/dispatcher/supervisor.py", line 313, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_d7c4062c8e48ef4dca373feb7285bdfb014e59b7/teuthology/dispatcher/supervisor.py", line 378, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_d7c4062c8e48ef4dca373feb7285bdfb014e59b7/teuthology/misc.py", line 1368, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_d7c4062c8e48ef4dca373feb7285bdfb014e59b7/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_d7c4062c8e48ef4dca373feb7285bdfb014e59b7/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_d7c4062c8e48ef4dca373feb7285bdfb014e59b7/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi063
2023-04-07T08:34:53.793 INFO:teuthology.kill:No teuthology processes running
2023-04-07T08:34:54.607 INFO:teuthology.kill:Nuking machines: ['smithi027', 'smithi063']
2023-04-07T08:34:55.293 INFO:teuthology.kill:2023-04-07 08:34:55,292.292 INFO:teuthology.nuke:targets:
2023-04-07T08:34:55.293 INFO:teuthology.kill:  smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBDWTxtSpO6JsMJqqp4VNmz9XRlm0LmpDop+BUxRzTPPDPw4+pLPeDS6Se3/L+3uDhZrt/JV3lksYbME3bodO7E=
2023-04-07T08:34:55.294 INFO:teuthology.kill:  smithi063.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6FneSdsumoIDgEn2g7lzW9aG6mUMboUMsmFkppJC8LYP5XqCTDKJajZDWQjAuk0f6oV9smEjqQDLmRhxPz8MyZ6StW5hdnQB5LfpMu2mzb8Fjs2m445EtuWN7aPra9a70p06zoVMZH3nVSSBDXjsCS7mDqDYdKsWg8wnYo8esYNjgGf1oxxgsv+XzOpc7MtSZabSdL/Puu1hK5kv8hCrF+DBZd9MKv0/cExP5O8c25TvbP/Tzzce7hyfkaedFYjxRdgKeTY+SSLEaqXGg+7dTB84Z5QJIZj2TUqNR6+NL/sua6qCHo/FvTUB+WqdXYJ3ZCaG+kbF+axxZsL3OaOX59OYJFHDwAeJvR+OYElrg+zEza0c4tmkdpA1jrnJhNgX2Mt4SZ+TWQkor1/V4CPGU7uUHQVeIf0ZwH6G1CcP+li+KR/nSf61II9v/mm/3IzID8ObyHpckOlM9QvPbWIR0rrNjTv4hyocw8JDD0nKpTzGUpB8whMrzNIPa1vi8tC8=
2023-04-07T08:34:55.294 INFO:teuthology.kill:2023-04-07 08:34:55,292.292 INFO:teuthology.nuke:Checking targets against current locks
2023-04-07T08:34:55.353 INFO:teuthology.kill:2023-04-07 08:34:55,353.353 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-04-07T08:34:55.374 INFO:teuthology.kill:2023-04-07 08:34:55,374.374 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-04-07T08:34:55.425 INFO:teuthology.kill:2023-04-07 08:34:55,425.425 INFO:teuthology.orchestra.console:Power off smithi027
2023-04-07T08:34:55.463 INFO:teuthology.kill:2023-04-07 08:34:55,463.463 INFO:teuthology.orchestra.console:Power off smithi063
2023-04-07T08:35:03.831 INFO:teuthology.kill:2023-04-07 08:35:03,830.830 INFO:teuthology.orchestra.console:Power off for smithi063 completed
2023-04-07T08:35:03.991 INFO:teuthology.kill:2023-04-07 08:35:03,991.991 INFO:teuthology.lock.ops:unlocked: smithi063.front.sepia.ceph.com
2023-04-07T08:35:07.913 INFO:teuthology.kill:2023-04-07 08:35:07,913.913 INFO:teuthology.orchestra.console:Power off for smithi027 completed
2023-04-07T08:35:08.066 INFO:teuthology.kill:2023-04-07 08:35:08,065.065 INFO:teuthology.lock.ops:unlocked: smithi027.front.sepia.ceph.com
2023-04-07T08:37:08.317 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-04-07T08:37:08.435 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi027 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-04-06_21:39:07-rbd-wip-yuri10-testing-2023-04-06-0846-pacific-distro-default-smithi/7234602
2023-04-07T08:37:08.435 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi063 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-04-06_21:39:07-rbd-wip-yuri10-testing-2023-04-06-0846-pacific-distro-default-smithi/7234601