2023-04-06T16:21:51.186 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae
2023-04-06T16:21:51.327 INFO:teuthology.lock.ops:Start node 'smithi073.front.sepia.ceph.com' reimaging
2023-04-06T16:21:51.328 INFO:teuthology.lock.ops:Updating [smithi073.front.sepia.ceph.com]: reset os type and version on server
2023-04-06T16:21:51.328 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server
2023-04-06T16:21:51.352 INFO:teuthology.lock.ops:Node 'smithi073.front.sepia.ceph.com' reimaging is complete
2023-04-06T16:21:51.352 INFO:teuthology.lock.ops:Start node 'smithi078.front.sepia.ceph.com' reimaging
2023-04-06T16:21:51.353 INFO:teuthology.lock.ops:Updating [smithi078.front.sepia.ceph.com]: reset os type and version on server
2023-04-06T16:21:51.353 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server
2023-04-06T16:21:51.376 INFO:teuthology.lock.ops:Node 'smithi078.front.sepia.ceph.com' reimaging is complete
2023-04-06T16:21:51.529 INFO:teuthology.provision.fog.smithi073:Scheduling deploy of centos 8.stream
2023-04-06T16:21:51.535 INFO:teuthology.provision.fog.smithi078:Scheduling deploy of centos 8.stream
2023-04-06T16:21:51.748 INFO:teuthology.orchestra.console:Power off smithi073
2023-04-06T16:21:51.762 INFO:teuthology.orchestra.console:Power off smithi078
2023-04-06T16:22:00.124 INFO:teuthology.orchestra.console:Power off for smithi073 completed
2023-04-06T16:22:00.225 INFO:teuthology.orchestra.console:Power on smithi073
2023-04-06T16:22:04.247 INFO:teuthology.orchestra.console:Power off for smithi078 completed
2023-04-06T16:22:04.348 INFO:teuthology.orchestra.console:Power on smithi078
2023-04-06T16:22:08.605 INFO:teuthology.orchestra.console:Power on for smithi078 completed
2023-04-06T16:22:08.693 INFO:teuthology.orchestra.console:Power on for smithi073 completed
2023-04-06T16:22:08.707 INFO:teuthology.provision.fog.smithi078:Waiting for deploy to finish
2023-04-06T16:22:08.795 INFO:teuthology.provision.fog.smithi073:Waiting for deploy to finish
2023-04-06T16:24:57.701 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.73
2023-04-06T16:25:06.757 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.73
2023-04-06T16:25:12.965 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.78
2023-04-06T16:25:15.813 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.73
2023-04-06T16:25:22.025 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.78
2023-04-06T16:25:24.869 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.73
2023-04-06T16:25:31.113 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.78
2023-04-06T16:25:33.925 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.73
2023-04-06T16:25:40.197 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.78
2023-04-06T16:25:42.981 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.73
2023-04-06T16:25:49.253 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.78
2023-04-06T16:25:52.037 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.73
2023-04-06T16:26:55.313 WARNING:teuthology.provision.fog:timed out
2023-04-06T16:26:58.045 WARNING:teuthology.provision.fog:timed out
2023-04-06T16:27:01.660 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-04-06T16:27:01.936 INFO:teuthology.provision.fog.smithi078:Node is ready
2023-04-06T16:27:01.952 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078.front.sepia.ceph.com
2023-04-06T16:27:02.009 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78 smithi078.front.sepia.ceph.com smithi078
2023-04-06T16:27:02.379 INFO:teuthology.provision.fog.smithi078:Deploy complete!
2023-04-06T16:27:04.430 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-04-06T16:27:04.672 INFO:teuthology.provision.fog.smithi073:Node is ready
2023-04-06T16:27:04.730 INFO:teuthology.orchestra.run.smithi073.stdout:smithi073.front.sepia.ceph.com
2023-04-06T16:27:04.787 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73 smithi073.front.sepia.ceph.com smithi073
2023-04-06T16:27:05.139 INFO:teuthology.provision.fog.smithi073:Deploy complete!
2023-04-06T16:27:05.307 INFO:teuthology.lock.ops:Checking smithi073.front.sepia.ceph.com
2023-04-06T16:27:05.308 INFO:teuthology.lock.ops:New key found. Updating...
2023-04-06T16:27:05.329 INFO:teuthology.lock.ops:Checking smithi078.front.sepia.ceph.com
2023-04-06T16:27:05.330 INFO:teuthology.lock.ops:New key found. Updating...
2023-04-06T16:27:05.355 INFO:teuthology.lock.ops:Updating [smithi073.front.sepia.ceph.com]: set os type and version on server
2023-04-06T16:27:05.356 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com'
2023-04-06T16:27:05.545 INFO:teuthology.orchestra.run.smithi073.stdout:x86_64
2023-04-06T16:27:05.603 INFO:teuthology.orchestra.run.smithi073.stdout:NAME="CentOS Stream"
2023-04-06T16:27:05.603 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION="8"
2023-04-06T16:27:05.603 INFO:teuthology.orchestra.run.smithi073.stdout:ID="centos"
2023-04-06T16:27:05.603 INFO:teuthology.orchestra.run.smithi073.stdout:ID_LIKE="rhel fedora"
2023-04-06T16:27:05.604 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION_ID="8"
2023-04-06T16:27:05.604 INFO:teuthology.orchestra.run.smithi073.stdout:PLATFORM_ID="platform:el8"
2023-04-06T16:27:05.604 INFO:teuthology.orchestra.run.smithi073.stdout:PRETTY_NAME="CentOS Stream 8"
2023-04-06T16:27:05.604 INFO:teuthology.orchestra.run.smithi073.stdout:ANSI_COLOR="0;31"
2023-04-06T16:27:05.604 INFO:teuthology.orchestra.run.smithi073.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-04-06T16:27:05.604 INFO:teuthology.orchestra.run.smithi073.stdout:HOME_URL="https://centos.org/"
2023-04-06T16:27:05.604 INFO:teuthology.orchestra.run.smithi073.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-04-06T16:27:05.605 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-04-06T16:27:05.605 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-04-06T16:27:05.606 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server
2023-04-06T16:27:05.629 INFO:teuthology.lock.ops:Updating [smithi078.front.sepia.ceph.com]: set os type and version on server
2023-04-06T16:27:05.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com'
2023-04-06T16:27:05.830 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64
2023-04-06T16:27:05.892 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Stream"
2023-04-06T16:27:05.892 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="8"
2023-04-06T16:27:05.892 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos"
2023-04-06T16:27:05.893 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora"
2023-04-06T16:27:05.893 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="8"
2023-04-06T16:27:05.893 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el8"
2023-04-06T16:27:05.893 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Stream 8"
2023-04-06T16:27:05.893 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31"
2023-04-06T16:27:05.894 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-04-06T16:27:05.894 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/"
2023-04-06T16:27:05.894 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-04-06T16:27:05.894 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-04-06T16:27:05.894 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-04-06T16:27:05.895 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server
2023-04-06T16:27:06.008 INFO:teuthology.dispatcher.supervisor:Running job 7234311
2023-04-06T16:27:06.009 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-04-06_15:37:58-rados-wip-yuri3-testing-2023-04-04-0833-pacific-distro-default-smithi/7234311 --name yuriw-2023-04-06_15:37:58-rados-wip-yuri3-testing-2023-04-04-0833-pacific-distro-default-smithi --description rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/yuriw-2023-04-06_15:37:58-rados-wip-yuri3-testing-2023-04-04-0833-pacific-distro-default-smithi/7234311/orig.config.yaml
2023-04-06T16:27:06.013 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-04-06_15:37:58-rados-wip-yuri3-testing-2023-04-04-0833-pacific-distro-default-smithi/7234311
2023-04-06T16:27:06.014 INFO:teuthology.dispatcher.supervisor:Job PID: 21471
2023-04-06T16:27:06.014 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-04-07T04:28:02.725 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-04-07T04:28:02.936 INFO:teuthology.kill:Killing Pids: {21471}
2023-04-07T04:28:03.812 INFO:teuthology.task.internal:roles: ubuntu@smithi073.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-04-07T04:28:03.812 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-04-07T04:28:03.813 INFO:teuthology.misc:Compressing logs...
2023-04-07T04:28:03.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com'
2023-04-07T04:28:04.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com'
2023-04-07T04:28:04.036 INFO:teuthology.orchestra.run.smithi073.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-04-07T04:28:19.083 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner
2023-04-07T04:28:19.085 ERROR:paramiko.transport:Traceback (most recent call last):
2023-04-07T04:28:19.086 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-04-07T04:28:19.086 ERROR:paramiko.transport:    buf = self.packetizer.readline(timeout)
2023-04-07T04:28:19.087 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline
2023-04-07T04:28:19.087 ERROR:paramiko.transport:    buf += self._read_timeout(timeout)
2023-04-07T04:28:19.088 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 622, in _read_timeout
2023-04-07T04:28:19.088 ERROR:paramiko.transport:    raise socket.timeout()
2023-04-07T04:28:19.088 ERROR:paramiko.transport:socket.timeout
2023-04-07T04:28:19.089 ERROR:paramiko.transport:
2023-04-07T04:28:19.089 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2023-04-07T04:28:19.090 ERROR:paramiko.transport:
2023-04-07T04:28:19.090 ERROR:paramiko.transport:Traceback (most recent call last):
2023-04-07T04:28:19.090 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2093, in run
2023-04-07T04:28:19.091 ERROR:paramiko.transport:    self._check_banner()
2023-04-07T04:28:19.091 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2274, in _check_banner
2023-04-07T04:28:19.092 ERROR:paramiko.transport:    raise SSHException(
2023-04-07T04:28:19.092 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-04-07T04:28:19.093 ERROR:paramiko.transport:
2023-04-07T04:28:19.093 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/dispatcher/supervisor.py", line 313, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/dispatcher/supervisor.py", line 378, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/misc.py", line 1368, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi078
2023-04-07T04:28:19.147 INFO:teuthology.kill:No teuthology processes running
2023-04-07T04:28:19.986 INFO:teuthology.kill:Nuking machines: ['smithi073', 'smithi078']
2023-04-07T04:28:20.567 INFO:teuthology.kill:2023-04-07 04:28:20,566.566 INFO:teuthology.nuke:targets:
2023-04-07T04:28:20.567 INFO:teuthology.kill:  smithi073.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC01XC58lZgG9DBpEyfGbtk2J00Zdgxj9DNWwTpHmEsD7my3amhb0QtlBXF8p4QvTeicB/jtcjKi7YPVMkdz2K27oFMujWh8MPExv/nhQs/2bRo2wudLfdSJHFztGfOtqoXyq1NkOIC9K44lgDb6afIqt9F+MNLtSMRN6rd5ryVi2A4hRMA4ktSq0sFCXx63Q2IKcBpJaaJflfx5h9hyw3if+yLnafLGNzK/RKn7L72DVE0mjZyBA0Cwo+bqSZ4c3IXkONgCEGQKo03VlzbF2joFSUa/o7X9IoaVaNJNRIo1XJRTG41M3bg2tf9UJo6HewYBaR5cPtCbqVLi9gKXyvarp1FATnYS+gWL0WkPNFpkFDzouLhWWByomj2Cd/YY3fUcrjuEj8S2ixhuULDEi8iKUcWp00zy2jhwjUc3kP4mZzLaVE/fkI3a8Z0I8FzTH8PTb+c+i/f7Q+CSr5NvUcS/BUkpOY/3Trs3JRotMeQquwV2gWVffF8pvT8VQgqRI8=
2023-04-07T04:28:20.568 INFO:teuthology.kill:  smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFrX481POzzpit4ngADNDtaL7eYmYB2Ab5rHizk5SmDh4ulHAQ8z1qn0U5V6lk0d/6ytN+YDKq5OR3Cv1CX4ffY=
2023-04-07T04:28:20.568 INFO:teuthology.kill:2023-04-07 04:28:20,566.566 INFO:teuthology.nuke:Checking targets against current locks
2023-04-07T04:28:20.624 INFO:teuthology.kill:2023-04-07 04:28:20,624.624 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-04-07T04:28:20.639 INFO:teuthology.kill:2023-04-07 04:28:20,639.639 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-04-07T04:28:20.694 INFO:teuthology.kill:2023-04-07 04:28:20,693.693 INFO:teuthology.orchestra.console:Power off smithi073
2023-04-07T04:28:20.711 INFO:teuthology.kill:2023-04-07 04:28:20,711.711 INFO:teuthology.orchestra.console:Power off smithi078
2023-04-07T04:28:29.065 INFO:teuthology.kill:2023-04-07 04:28:29,065.065 INFO:teuthology.orchestra.console:Power off for smithi073 completed
2023-04-07T04:28:29.222 INFO:teuthology.kill:2023-04-07 04:28:29,222.222 INFO:teuthology.lock.ops:unlocked: smithi073.front.sepia.ceph.com
2023-04-07T04:28:33.183 INFO:teuthology.kill:2023-04-07 04:28:33,182.182 INFO:teuthology.orchestra.console:Power off for smithi078 completed
2023-04-07T04:28:33.337 INFO:teuthology.kill:2023-04-07 04:28:33,337.337 INFO:teuthology.lock.ops:unlocked: smithi078.front.sepia.ceph.com
2023-04-07T04:30:33.704 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15