2023-05-17T19:41:08.028 INFO:root:teuthology version: 0.0.1.dev98+g69caab8
2023-05-17T19:41:08.175 INFO:teuthology.lock.ops:Start node 'smithi130.front.sepia.ceph.com' reimaging
2023-05-17T19:41:08.176 INFO:teuthology.lock.ops:Updating [smithi130.front.sepia.ceph.com]: reset os type and version on server
2023-05-17T19:41:08.176 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server
2023-05-17T19:41:08.196 INFO:teuthology.lock.ops:Node 'smithi130.front.sepia.ceph.com' reimaging is complete
2023-05-17T19:41:08.196 INFO:teuthology.lock.ops:Start node 'smithi155.front.sepia.ceph.com' reimaging
2023-05-17T19:41:08.196 INFO:teuthology.lock.ops:Updating [smithi155.front.sepia.ceph.com]: reset os type and version on server
2023-05-17T19:41:08.196 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server
2023-05-17T19:41:08.212 INFO:teuthology.lock.ops:Node 'smithi155.front.sepia.ceph.com' reimaging is complete
2023-05-17T19:41:08.354 INFO:teuthology.provision.fog.smithi130:Scheduling deploy of centos 8.stream
2023-05-17T19:41:08.376 INFO:teuthology.provision.fog.smithi155:Scheduling deploy of centos 8.stream
2023-05-17T19:41:08.642 INFO:teuthology.orchestra.console:Power off smithi155
2023-05-17T19:41:08.657 INFO:teuthology.orchestra.console:Power off smithi130
2023-05-17T19:41:17.018 INFO:teuthology.orchestra.console:Power off for smithi155 completed
2023-05-17T19:41:17.119 INFO:teuthology.orchestra.console:Power on smithi155
2023-05-17T19:41:21.139 INFO:teuthology.orchestra.console:Power off for smithi130 completed
2023-05-17T19:41:21.240 INFO:teuthology.orchestra.console:Power on smithi130
2023-05-17T19:41:25.488 INFO:teuthology.orchestra.console:Power on for smithi155 completed
2023-05-17T19:41:25.493 INFO:teuthology.orchestra.console:Power on for smithi130 completed
2023-05-17T19:41:25.589 INFO:teuthology.provision.fog.smithi155:Waiting for deploy to finish
2023-05-17T19:41:25.594 INFO:teuthology.provision.fog.smithi130:Waiting for deploy to finish
2023-05-17T19:45:01.552 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.155
2023-05-17T19:45:10.608 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.155
2023-05-17T19:45:19.664 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.155
2023-05-17T19:45:28.720 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.155
2023-05-17T19:45:37.776 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.155
2023-05-17T19:45:46.832 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.155
2023-05-17T19:46:47.856 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.130
2023-05-17T19:46:52.843 WARNING:teuthology.provision.fog:timed out
2023-05-17T19:46:56.912 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.130
2023-05-17T19:46:59.067 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-17T19:46:59.319 INFO:teuthology.provision.fog.smithi155:Node is ready
2023-05-17T19:46:59.337 INFO:teuthology.orchestra.run.smithi155.stdout:smithi155.front.sepia.ceph.com
2023-05-17T19:46:59.394 INFO:teuthology.orchestra.run.smithi155.stdout:172.21.15.155 smithi155.front.sepia.ceph.com smithi155
2023-05-17T19:46:59.719 INFO:teuthology.provision.fog.smithi155:Deploy complete!
2023-05-17T19:47:05.968 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.130
2023-05-17T19:47:15.028 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.130
2023-05-17T19:47:24.112 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.130
2023-05-17T19:47:33.168 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.130
2023-05-17T19:47:42.224 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.130
2023-05-17T19:48:48.270 WARNING:teuthology.provision.fog:timed out
2023-05-17T19:48:54.497 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-17T19:48:54.765 INFO:teuthology.provision.fog.smithi130:Node is ready
2023-05-17T19:48:54.783 INFO:teuthology.orchestra.run.smithi130.stdout:smithi130.front.sepia.ceph.com
2023-05-17T19:48:54.842 INFO:teuthology.orchestra.run.smithi130.stdout:172.21.15.130 smithi130.front.sepia.ceph.com smithi130
2023-05-17T19:48:55.169 INFO:teuthology.provision.fog.smithi130:Deploy complete!
2023-05-17T19:48:55.343 INFO:teuthology.lock.ops:Checking smithi130.front.sepia.ceph.com
2023-05-17T19:48:55.344 INFO:teuthology.lock.ops:New key found. Updating...
2023-05-17T19:48:55.371 INFO:teuthology.lock.ops:Checking smithi155.front.sepia.ceph.com
2023-05-17T19:48:55.371 INFO:teuthology.lock.ops:New key found. Updating...
2023-05-17T19:48:55.389 INFO:teuthology.lock.ops:Updating [smithi130.front.sepia.ceph.com]: set os type and version on server
2023-05-17T19:48:55.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com'
2023-05-17T19:48:55.579 INFO:teuthology.orchestra.run.smithi130.stdout:x86_64
2023-05-17T19:48:55.635 INFO:teuthology.orchestra.run.smithi130.stdout:NAME="CentOS Stream"
2023-05-17T19:48:55.636 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION="8"
2023-05-17T19:48:55.636 INFO:teuthology.orchestra.run.smithi130.stdout:ID="centos"
2023-05-17T19:48:55.636 INFO:teuthology.orchestra.run.smithi130.stdout:ID_LIKE="rhel fedora"
2023-05-17T19:48:55.636 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_ID="8"
2023-05-17T19:48:55.637 INFO:teuthology.orchestra.run.smithi130.stdout:PLATFORM_ID="platform:el8"
2023-05-17T19:48:55.637 INFO:teuthology.orchestra.run.smithi130.stdout:PRETTY_NAME="CentOS Stream 8"
2023-05-17T19:48:55.637 INFO:teuthology.orchestra.run.smithi130.stdout:ANSI_COLOR="0;31"
2023-05-17T19:48:55.637 INFO:teuthology.orchestra.run.smithi130.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-05-17T19:48:55.637 INFO:teuthology.orchestra.run.smithi130.stdout:HOME_URL="https://centos.org/"
2023-05-17T19:48:55.637 INFO:teuthology.orchestra.run.smithi130.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-05-17T19:48:55.638 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-05-17T19:48:55.638 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-05-17T19:48:55.639 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server
2023-05-17T19:48:55.663 INFO:teuthology.lock.ops:Updating [smithi155.front.sepia.ceph.com]: set os type and version on server
2023-05-17T19:48:55.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com'
2023-05-17T19:48:55.927 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64
2023-05-17T19:48:55.985 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="CentOS Stream"
2023-05-17T19:48:55.985 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="8"
2023-05-17T19:48:55.985 INFO:teuthology.orchestra.run.smithi155.stdout:ID="centos"
2023-05-17T19:48:55.986 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="rhel fedora"
2023-05-17T19:48:55.986 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="8"
2023-05-17T19:48:55.986 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el8"
2023-05-17T19:48:55.986 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="CentOS Stream 8"
2023-05-17T19:48:55.986 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31"
2023-05-17T19:48:55.986 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-05-17T19:48:55.986 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://centos.org/"
2023-05-17T19:48:55.987 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-05-17T19:48:55.987 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-05-17T19:48:55.987 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-05-17T19:48:55.987 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server
2023-05-17T19:48:56.108 INFO:teuthology.dispatcher.supervisor:Running job 7276765
2023-05-17T19:48:56.108 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-05-17_19:39:18-rados-wip-yuri5-testing-2023-05-09-1324-pacific-distro-default-smithi/7276765 --name yuriw-2023-05-17_19:39:18-rados-wip-yuri5-testing-2023-05-09-1324-pacific-distro-default-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/yuriw-2023-05-17_19:39:18-rados-wip-yuri5-testing-2023-05-09-1324-pacific-distro-default-smithi/7276765/orig.config.yaml
2023-05-17T19:48:56.111 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-05-17_19:39:18-rados-wip-yuri5-testing-2023-05-09-1324-pacific-distro-default-smithi/7276765
2023-05-17T19:48:56.112 INFO:teuthology.dispatcher.supervisor:Job PID: 25590
2023-05-17T19:48:56.113 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-05-18T07:49:53.751 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-05-18T07:49:53.929 INFO:teuthology.kill:Killing Pids: {25590}
2023-05-18T07:49:54.785 INFO:teuthology.task.internal:roles: ubuntu@smithi130.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-05-18T07:49:54.786 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-05-18T07:49:54.786 INFO:teuthology.misc:Compressing logs...
2023-05-18T07:49:54.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com'
2023-05-18T07:49:54.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com'
2023-05-18T07:49:55.011 INFO:teuthology.orchestra.run.smithi130.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-05-18T07:50:09.999 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner
2023-05-18T07:50:10.002 ERROR:paramiko.transport:Traceback (most recent call last):
2023-05-18T07:50:10.002 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-05-18T07:50:10.002 ERROR:paramiko.transport:    buf = self.packetizer.readline(timeout)
2023-05-18T07:50:10.002 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline
2023-05-18T07:50:10.003 ERROR:paramiko.transport:    buf += self._read_timeout(timeout)
2023-05-18T07:50:10.003 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 622, in _read_timeout
2023-05-18T07:50:10.003 ERROR:paramiko.transport:    raise socket.timeout()
2023-05-18T07:50:10.003 ERROR:paramiko.transport:socket.timeout
2023-05-18T07:50:10.003 ERROR:paramiko.transport:
2023-05-18T07:50:10.004 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2023-05-18T07:50:10.004 ERROR:paramiko.transport:
2023-05-18T07:50:10.004 ERROR:paramiko.transport:Traceback (most recent call last):
2023-05-18T07:50:10.005 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2093, in run
2023-05-18T07:50:10.005 ERROR:paramiko.transport:    self._check_banner()
2023-05-18T07:50:10.005 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2274, in _check_banner
2023-05-18T07:50:10.005 ERROR:paramiko.transport:    raise SSHException(
2023-05-18T07:50:10.005 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-05-18T07:50:10.006 ERROR:paramiko.transport:
2023-05-18T07:50:10.006 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/dispatcher/supervisor.py", line 313, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/dispatcher/supervisor.py", line 378, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/misc.py", line 1368, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi155
2023-05-18T07:50:10.075 INFO:teuthology.kill:No teuthology processes running
2023-05-18T07:50:11.185 INFO:teuthology.kill:Nuking machines: ['smithi130', 'smithi155']
2023-05-18T07:50:11.915 INFO:teuthology.kill:2023-05-18 07:50:11,914.914 INFO:teuthology.nuke:targets:
2023-05-18T07:50:11.916 INFO:teuthology.kill:  smithi130.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDma3OpFfmVrl/1dd2p/9PtqhxZLO/9lExBHdlODN1ZDOqBGOZk3vO+c3zx8pQ4u9g2jkG/UAJ133RoYQBjTKOEb5bQSAHMa7yk++oQDGuMtmn2HHKxQ1I0DuJiNVcu2InakNzDfi3HgokGqirWjq+VZvrqxCFaF/NMWeISE9EPLc2DT2P9aBCf1tS9JVjVvBlRh9WdCuTm5Wl2hFvg4Gu7s8rUqWV/Pj1TUsdNalfXOOEgw/IV5Y7PJvMPP+Z+qYs0B986DrehqJDsU9D1QkUBwE5sQQdzhow5d1AsTZKENV/uOYzejsI9qc6vXFxIEuxicUsz1K0bK7HkVgeNsLf4bKKk0H70sBt8deQbD9Q9+6CtTYFWfoRoBlkV7eRJnkvJ2rnc5PFWlOnqjsLZK0wjHhrHFOetZjR5AL0ZmUfY9mkxykQd7EiNXX5fsz3iYoZ5ytSpGBrn2Yc40VEPDxyOecdH3PoNYmmBbbG9euWNp19lD5YA/UHw9/4SZGsZ23U=
2023-05-18T07:50:11.917 INFO:teuthology.kill:  smithi155.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDF72aTmGgct99bL50925mYq8eEIJRP9TQspUGR/QiPABiFwsXt1JPIDRzfVhfASjVvRqy//MFxRrUvO0OZbt0wjbJ8FM/hSiB9PKkSIEmj9x1mtWc0iUOi+Kwo6vWmCjXVAF7OzNfG+QiPKzcqQCXa1X9FhzCvSWOZiGrXFLvXI5igKgZemOF+JWzXevE5s6HeA3yO5ETo7GgLPzXi0WcPr20pGfKi0DVeQY8a9nhN9LBr8bbDMjKEfX3hsCrV8lH+FqZyyxMAX7PS6Z4fm9osLw1Dx8v3iRkBfV/FlUaLvWSDw0eyAf97pmjEnH50PzQcbj5HRoc3hBmOgIH/+ctkugObcuwo0GsKyYQqMKl1G5Mtm3vrAli/FgTbF9cU/jNccCqgDt2oUCV7yInFZQDDVG8u2gQ+I532+SgxvJcT6lqBA1dlV1qxD3GXcbHNwtMiHdY20tTMTmbfkgH+4lsLJ5CfgHryeIaMQ0jIG8IROnuEcB4Zhaxxsd/HrKd9C/8=
2023-05-18T07:50:11.919 INFO:teuthology.kill:2023-05-18 07:50:11,915.915 INFO:teuthology.nuke:Checking targets against current locks
2023-05-18T07:50:11.985 INFO:teuthology.kill:2023-05-18 07:50:11,985.985 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-18T07:50:12.014 INFO:teuthology.kill:2023-05-18 07:50:12,013.013 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-18T07:50:12.095 INFO:teuthology.kill:2023-05-18 07:50:12,086.086 INFO:teuthology.orchestra.console:Power off smithi130
2023-05-18T07:50:12.154 INFO:teuthology.kill:2023-05-18 07:50:12,154.154 INFO:teuthology.orchestra.console:Power off smithi155
2023-05-18T07:50:20.508 INFO:teuthology.kill:2023-05-18 07:50:20,507.507 INFO:teuthology.orchestra.console:Power off for smithi130 completed
2023-05-18T07:50:20.551 INFO:teuthology.kill:2023-05-18 07:50:20,549.549 INFO:teuthology.orchestra.console:Power off for smithi155 completed
2023-05-18T07:50:20.673 INFO:teuthology.kill:2023-05-18 07:50:20,671.671 INFO:teuthology.lock.ops:unlocked: smithi130.front.sepia.ceph.com
2023-05-18T07:50:20.703 INFO:teuthology.kill:2023-05-18 07:50:20,702.702 INFO:teuthology.lock.ops:unlocked: smithi155.front.sepia.ceph.com
2023-05-18T07:52:21.147 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-05-18T07:52:21.239 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi130 but it was locked by another job: /home/teuthworker/archive/dis-2023-05-18_07:24:53-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277145
2023-05-18T07:52:21.240 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi155 but it was locked by another job: /home/teuthworker/archive/dis-2023-05-18_07:24:53-rbd-wip-rbd-reef-backports-2-distro-default-smithi/7277145