2023-04-24T12:24:52.034 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae
2023-04-24T12:24:52.187 INFO:teuthology.lock.ops:Start node 'smithi019.front.sepia.ceph.com' reimaging
2023-04-24T12:24:52.188 INFO:teuthology.lock.ops:Updating [smithi019.front.sepia.ceph.com]: reset os type and version on server
2023-04-24T12:24:52.189 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server
2023-04-24T12:24:52.217 INFO:teuthology.lock.ops:Node 'smithi019.front.sepia.ceph.com' reimaging is complete
2023-04-24T12:24:52.218 INFO:teuthology.lock.ops:Start node 'smithi115.front.sepia.ceph.com' reimaging
2023-04-24T12:24:52.218 INFO:teuthology.lock.ops:Updating [smithi115.front.sepia.ceph.com]: reset os type and version on server
2023-04-24T12:24:52.218 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server
2023-04-24T12:24:52.242 INFO:teuthology.lock.ops:Node 'smithi115.front.sepia.ceph.com' reimaging is complete
2023-04-24T12:24:52.377 INFO:teuthology.provision.fog.smithi019:Scheduling deploy of centos 8.stream
2023-04-24T12:24:52.385 INFO:teuthology.provision.fog.smithi115:Scheduling deploy of centos 8.stream
2023-04-24T12:24:52.597 INFO:teuthology.orchestra.console:Power off smithi115
2023-04-24T12:24:52.643 INFO:teuthology.orchestra.console:Power off smithi019
2023-04-24T12:24:52.785 INFO:teuthology.orchestra.console:Power off for smithi019 completed
2023-04-24T12:24:52.886 INFO:teuthology.orchestra.console:Power on smithi019
2023-04-24T12:24:57.144 INFO:teuthology.orchestra.console:Power on for smithi019 completed
2023-04-24T12:24:57.246 INFO:teuthology.provision.fog.smithi019:Waiting for deploy to finish
2023-04-24T12:25:05.090 INFO:teuthology.orchestra.console:Power off for smithi115 completed
2023-04-24T12:25:05.191 INFO:teuthology.orchestra.console:Power on smithi115
2023-04-24T12:25:09.448 INFO:teuthology.orchestra.console:Power on for smithi115 completed
2023-04-24T12:25:09.549 INFO:teuthology.provision.fog.smithi115:Waiting for deploy to finish
2023-04-24T12:27:46.021 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.19
2023-04-24T12:27:55.077 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.19
2023-04-24T12:27:55.529 ERROR:teuthology.orchestra.connection:Error authenticating with smithi115.front.sepia.ceph.com: Authentication failed.
2023-04-24T12:28:04.133 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.19
2023-04-24T12:28:13.189 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.19
2023-04-24T12:28:22.245 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.19
2023-04-24T12:28:56.532 WARNING:teuthology.provision.fog:timed out
2023-04-24T12:29:28.247 WARNING:teuthology.provision.fog:timed out
2023-04-24T12:29:34.560 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-04-24T12:29:34.851 INFO:teuthology.provision.fog.smithi019:Node is ready
2023-04-24T12:29:34.870 INFO:teuthology.orchestra.run.smithi019.stdout:smithi019.front.sepia.ceph.com
2023-04-24T12:29:34.930 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19 smithi019.front.sepia.ceph.com smithi019
2023-04-24T12:29:35.294 INFO:teuthology.provision.fog.smithi019:Deploy complete!
2023-04-24T12:30:02.536 WARNING:teuthology.provision.fog:timed out
2023-04-24T12:30:08.878 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-04-24T12:30:09.133 INFO:teuthology.provision.fog.smithi115:Node is ready
2023-04-24T12:30:09.190 INFO:teuthology.orchestra.run.smithi115.stdout:smithi115.front.sepia.ceph.com
2023-04-24T12:30:09.248 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115 smithi115.front.sepia.ceph.com smithi115
2023-04-24T12:30:09.602 INFO:teuthology.provision.fog.smithi115:Deploy complete!
2023-04-24T12:30:09.798 INFO:teuthology.lock.ops:Checking smithi019.front.sepia.ceph.com
2023-04-24T12:30:09.799 INFO:teuthology.lock.ops:New key found. Updating...
2023-04-24T12:30:09.827 INFO:teuthology.lock.ops:Checking smithi115.front.sepia.ceph.com
2023-04-24T12:30:09.828 INFO:teuthology.lock.ops:New key found. Updating...
2023-04-24T12:30:09.853 INFO:teuthology.lock.ops:Updating [smithi019.front.sepia.ceph.com]: set os type and version on server
2023-04-24T12:30:09.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-04-24T12:30:10.047 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64
2023-04-24T12:30:10.106 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="CentOS Stream"
2023-04-24T12:30:10.106 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="8"
2023-04-24T12:30:10.106 INFO:teuthology.orchestra.run.smithi019.stdout:ID="centos"
2023-04-24T12:30:10.107 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="rhel fedora"
2023-04-24T12:30:10.107 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="8"
2023-04-24T12:30:10.107 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el8"
2023-04-24T12:30:10.107 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="CentOS Stream 8"
2023-04-24T12:30:10.107 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31"
2023-04-24T12:30:10.107 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-04-24T12:30:10.107 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://centos.org/"
2023-04-24T12:30:10.108 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-04-24T12:30:10.108 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-04-24T12:30:10.108 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-04-24T12:30:10.109 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server
2023-04-24T12:30:10.135 INFO:teuthology.lock.ops:Updating [smithi115.front.sepia.ceph.com]: set os type and version on server
2023-04-24T12:30:10.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com'
2023-04-24T12:30:10.321 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64
2023-04-24T12:30:10.377 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="CentOS Stream"
2023-04-24T12:30:10.377 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="8"
2023-04-24T12:30:10.377 INFO:teuthology.orchestra.run.smithi115.stdout:ID="centos"
2023-04-24T12:30:10.377 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE="rhel fedora"
2023-04-24T12:30:10.377 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="8"
2023-04-24T12:30:10.378 INFO:teuthology.orchestra.run.smithi115.stdout:PLATFORM_ID="platform:el8"
2023-04-24T12:30:10.378 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="CentOS Stream 8"
2023-04-24T12:30:10.378 INFO:teuthology.orchestra.run.smithi115.stdout:ANSI_COLOR="0;31"
2023-04-24T12:30:10.378 INFO:teuthology.orchestra.run.smithi115.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-04-24T12:30:10.378 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://centos.org/"
2023-04-24T12:30:10.378 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-04-24T12:30:10.378 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-04-24T12:30:10.379 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-04-24T12:30:10.379 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server
2023-04-24T12:30:10.480 INFO:teuthology.dispatcher.supervisor:Running job 7249169
2023-04-24T12:30:10.480 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-04-23_18:59:17-orch-pacific-distro-default-smithi/7249169 --name yuriw-2023-04-23_18:59:17-orch-pacific-distro-default-smithi --description orch/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/yuriw-2023-04-23_18:59:17-orch-pacific-distro-default-smithi/7249169/orig.config.yaml
2023-04-24T12:30:10.484 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-04-23_18:59:17-orch-pacific-distro-default-smithi/7249169
2023-04-24T12:30:10.485 INFO:teuthology.dispatcher.supervisor:Job PID: 30231
2023-04-24T12:30:10.485 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-04-25T00:30:33.955 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-04-25T00:30:34.138 INFO:teuthology.kill:Killing Pids: {30231}
2023-04-25T00:30:35.006 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-04-25T00:30:35.007 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-04-25T00:30:35.007 INFO:teuthology.misc:Compressing logs...
2023-04-25T00:30:35.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-04-25T00:30:35.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com'
2023-04-25T00:30:35.194 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-04-25T00:30:50.255 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner
2023-04-25T00:30:50.300 ERROR:paramiko.transport:Traceback (most recent call last):
2023-04-25T00:30:50.300 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-04-25T00:30:50.300 ERROR:paramiko.transport:    buf = self.packetizer.readline(timeout)
2023-04-25T00:30:50.301 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline
2023-04-25T00:30:50.301 ERROR:paramiko.transport:    buf += self._read_timeout(timeout)
2023-04-25T00:30:50.301 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 622, in _read_timeout
2023-04-25T00:30:50.301 ERROR:paramiko.transport:    raise socket.timeout()
2023-04-25T00:30:50.302 ERROR:paramiko.transport:socket.timeout
2023-04-25T00:30:50.302 ERROR:paramiko.transport:
2023-04-25T00:30:50.302 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2023-04-25T00:30:50.302 ERROR:paramiko.transport:
2023-04-25T00:30:50.302 ERROR:paramiko.transport:Traceback (most recent call last):
2023-04-25T00:30:50.303 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2093, in run
2023-04-25T00:30:50.303 ERROR:paramiko.transport:    self._check_banner()
2023-04-25T00:30:50.303 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2274, in _check_banner
2023-04-25T00:30:50.303 ERROR:paramiko.transport:    raise SSHException(
2023-04-25T00:30:50.304 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-04-25T00:30:50.304 ERROR:paramiko.transport:
2023-04-25T00:30:50.304 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/dispatcher/supervisor.py", line 313, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/dispatcher/supervisor.py", line 378, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/misc.py", line 1368, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi115
2023-04-25T00:30:50.356 INFO:teuthology.kill:No teuthology processes running
2023-04-25T00:30:51.162 INFO:teuthology.kill:Nuking machines: ['smithi019', 'smithi115']
2023-04-25T00:30:51.789 INFO:teuthology.kill:2023-04-25 00:30:51,788.788 INFO:teuthology.nuke:targets:
2023-04-25T00:30:51.790 INFO:teuthology.kill:  smithi019.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE65YYmQ66dhFyqb+xtyMwbk3DDqY/4l0FDw0xzQ7sfvlitjkCk4H6fEmElrfKAOs9LQplvVGbkXSor15bQPQtpgPvJImRN65kovF67CLvN78rRKvpZ63xIDLrdkVfFbKXqpfVG3LoHBgIJ7RhCVIVjAsYDoxseGesmaJcXaDe08IZFVZnylrjY1NHMeDaBYX9x+sCU8OIzPCpSBYFjYrCSlsRBof3qkzI1PF1GaBlxcqA+83fDidnRZPmSK2WPlrA474174tWPeQAnumHDVZg6OcX0GQkALbb2rWpxvuh6+IbDxiIYRDFKb83lxmIsroBmJY6dURRvCe9gxf72mogjUJlXa2HrnIRLly4QuFRM6UWMJX2KApbEicdshqUsN+eeT8GIQZUAkc2qeJXBKMTGcmfH1UTzfmLCt6ew6TkMsCDp1IyJs0PgnJv0/tkMBLiUHN3Tx5I4OY81MWdlHELDeckrEdcqtzJmXphRsQH+3194hJCO0nXdNoM0yB1Ue8=
2023-04-25T00:30:51.790 INFO:teuthology.kill:  smithi115.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKJyf0yyskh4YAdovCApYXv5bzpJUy9MzJZ/TepfAYDfG/XMRgliJHDxjYIWvFzifp0SpnDWmshRkhbPZt0aw/52QqI9ytzGoqvK2bKqhJzUNd3o4n7c3KbMusFwT5bq7kxbmxyAlSynWkIvQDcxTHZgNxaMYq97PvyNgMrm7n6UwCCVvaV0Ow3fsjRgYpeDb/J8NIKaZEvrgECBJYLrGmfiqh0Bl0k35bfocvSTJb7a3oAKvHS+iLJtD4iVE2XhlEzZFyliDLnyjImeKeepRIaXnq/wPraKXLshewoT6ycvf2cIRcCZHLmE+D5xP5N3Y0deysrLvNAJJ9ycftBPRMfOZhKXztnK2/lhQyAKMDm/4fr+CFhyXRhFY3wdCgEOzSpGQIb/bMCXVgzK7lS2zfBM0zGBmOwnByOOJJxps8XXNVYw+auRiNtkIyqufbPhSeVc+gAAChgMT9aIkD47pHLPkWgYNf9NhAl/NCxwvO8cEsmlVhi4O3BbOIe3+45Ys=
2023-04-25T00:30:51.790 INFO:teuthology.kill:2023-04-25 00:30:51,789.789 INFO:teuthology.nuke:Checking targets against current locks
2023-04-25T00:30:51.846 INFO:teuthology.kill:2023-04-25 00:30:51,846.846 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-04-25T00:30:51.861 INFO:teuthology.kill:2023-04-25 00:30:51,861.861 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-04-25T00:30:51.915 INFO:teuthology.kill:2023-04-25 00:30:51,915.915 INFO:teuthology.orchestra.console:Power off smithi019
2023-04-25T00:30:51.954 INFO:teuthology.kill:2023-04-25 00:30:51,954.954 INFO:teuthology.orchestra.console:Power off smithi115
2023-04-25T00:31:04.404 INFO:teuthology.kill:2023-04-25 00:31:04,403.403 INFO:teuthology.orchestra.console:Power off for smithi019 completed
2023-04-25T00:31:04.435 INFO:teuthology.kill:2023-04-25 00:31:04,435.435 INFO:teuthology.orchestra.console:Power off for smithi115 completed
2023-04-25T00:31:04.565 INFO:teuthology.kill:2023-04-25 00:31:04,565.565 INFO:teuthology.lock.ops:unlocked: smithi019.front.sepia.ceph.com
2023-04-25T00:31:04.581 INFO:teuthology.kill:2023-04-25 00:31:04,580.580 INFO:teuthology.lock.ops:unlocked: smithi115.front.sepia.ceph.com
2023-04-25T00:33:04.823 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-04-25T00:33:04.924 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi019 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-04-24_23:35:26-smoke-pacific-release-distro-default-smithi/7250667