2023-05-22T10:22:14.676 INFO:root:teuthology version: 0.0.1.dev100+g01abd1e 2023-05-22T10:22:14.838 INFO:teuthology.lock.ops:Start node 'smithi150.front.sepia.ceph.com' reimaging 2023-05-22T10:22:14.839 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: reset os type and version on server 2023-05-22T10:22:14.839 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2023-05-22T10:22:14.863 INFO:teuthology.lock.ops:Node 'smithi150.front.sepia.ceph.com' reimaging is complete 2023-05-22T10:22:14.864 INFO:teuthology.lock.ops:Start node 'smithi171.front.sepia.ceph.com' reimaging 2023-05-22T10:22:14.864 INFO:teuthology.lock.ops:Updating [smithi171.front.sepia.ceph.com]: reset os type and version on server 2023-05-22T10:22:14.864 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server 2023-05-22T10:22:14.887 INFO:teuthology.lock.ops:Node 'smithi171.front.sepia.ceph.com' reimaging is complete 2023-05-22T10:22:15.024 INFO:teuthology.provision.fog.smithi150:Scheduling deploy of centos 8.stream 2023-05-22T10:22:15.040 INFO:teuthology.provision.fog.smithi171:Scheduling deploy of centos 8.stream 2023-05-22T10:22:15.232 INFO:teuthology.orchestra.console:Power off smithi171 2023-05-22T10:22:15.248 INFO:teuthology.orchestra.console:Power off smithi150 2023-05-22T10:22:23.608 INFO:teuthology.orchestra.console:Power off for smithi171 completed 2023-05-22T10:22:23.616 INFO:teuthology.orchestra.console:Power off for smithi150 completed 2023-05-22T10:22:23.710 INFO:teuthology.orchestra.console:Power on smithi171 2023-05-22T10:22:23.719 INFO:teuthology.orchestra.console:Power on smithi150 2023-05-22T10:22:32.092 INFO:teuthology.orchestra.console:Power on for smithi150 completed 2023-05-22T10:22:32.190 INFO:teuthology.orchestra.console:Power on for smithi171 completed 2023-05-22T10:22:32.194 INFO:teuthology.provision.fog.smithi150:Waiting for deploy to finish 2023-05-22T10:22:32.292 INFO:teuthology.provision.fog.smithi171:Waiting for deploy to finish 2023-05-22T10:25:21.104 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-05-22T10:25:21.106 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-05-22T10:25:30.192 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-05-22T10:25:30.193 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-05-22T10:25:39.248 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-05-22T10:25:39.249 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-05-22T10:25:48.304 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-05-22T10:25:48.305 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-05-22T10:25:57.360 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-05-22T10:25:57.361 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-05-22T10:26:06.416 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-05-22T10:26:06.417 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-05-22T10:26:15.472 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-05-22T10:26:15.473 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-05-22T10:27:21.531 WARNING:teuthology.provision.fog:timed out 2023-05-22T10:27:21.538 WARNING:teuthology.provision.fog:timed out 2023-05-22T10:27:27.749 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-05-22T10:27:27.824 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-05-22T10:27:28.016 INFO:teuthology.provision.fog.smithi150:Node is ready 2023-05-22T10:27:28.059 INFO:teuthology.provision.fog.smithi171:Node is ready 2023-05-22T10:27:28.076 INFO:teuthology.orchestra.run.smithi150.stdout:smithi150.front.sepia.ceph.com 2023-05-22T10:27:28.077 INFO:teuthology.orchestra.run.smithi171.stdout:smithi171.front.sepia.ceph.com 2023-05-22T10:27:28.137 INFO:teuthology.orchestra.run.smithi171.stdout:172.21.15.171 smithi171.front.sepia.ceph.com smithi171 2023-05-22T10:27:28.139 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150 smithi150.front.sepia.ceph.com smithi150 2023-05-22T10:27:28.459 INFO:teuthology.provision.fog.smithi171:Deploy complete! 2023-05-22T10:27:28.476 INFO:teuthology.provision.fog.smithi150:Deploy complete! 2023-05-22T10:27:28.656 INFO:teuthology.lock.ops:Checking smithi150.front.sepia.ceph.com 2023-05-22T10:27:28.657 INFO:teuthology.lock.ops:New key found. Updating... 2023-05-22T10:27:28.679 INFO:teuthology.lock.ops:Checking smithi171.front.sepia.ceph.com 2023-05-22T10:27:28.680 INFO:teuthology.lock.ops:New key found. Updating... 2023-05-22T10:27:28.705 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: set os type and version on server 2023-05-22T10:27:28.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-05-22T10:27:28.896 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2023-05-22T10:27:28.954 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream" 2023-05-22T10:27:28.954 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="8" 2023-05-22T10:27:28.954 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos" 2023-05-22T10:27:28.954 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora" 2023-05-22T10:27:28.954 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="8" 2023-05-22T10:27:28.955 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el8" 2023-05-22T10:27:28.955 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 8" 2023-05-22T10:27:28.955 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2023-05-22T10:27:28.955 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-05-22T10:27:28.955 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/" 2023-05-22T10:27:28.955 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-05-22T10:27:28.956 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-05-22T10:27:28.956 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-05-22T10:27:28.957 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2023-05-22T10:27:28.987 INFO:teuthology.lock.ops:Updating [smithi171.front.sepia.ceph.com]: set os type and version on server 2023-05-22T10:27:28.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2023-05-22T10:27:29.187 INFO:teuthology.orchestra.run.smithi171.stdout:x86_64 2023-05-22T10:27:29.245 INFO:teuthology.orchestra.run.smithi171.stdout:NAME="CentOS Stream" 2023-05-22T10:27:29.246 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION="8" 2023-05-22T10:27:29.246 INFO:teuthology.orchestra.run.smithi171.stdout:ID="centos" 2023-05-22T10:27:29.246 INFO:teuthology.orchestra.run.smithi171.stdout:ID_LIKE="rhel fedora" 2023-05-22T10:27:29.246 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION_ID="8" 2023-05-22T10:27:29.246 INFO:teuthology.orchestra.run.smithi171.stdout:PLATFORM_ID="platform:el8" 2023-05-22T10:27:29.246 INFO:teuthology.orchestra.run.smithi171.stdout:PRETTY_NAME="CentOS Stream 8" 2023-05-22T10:27:29.246 INFO:teuthology.orchestra.run.smithi171.stdout:ANSI_COLOR="0;31" 2023-05-22T10:27:29.247 INFO:teuthology.orchestra.run.smithi171.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-05-22T10:27:29.247 INFO:teuthology.orchestra.run.smithi171.stdout:HOME_URL="https://centos.org/" 2023-05-22T10:27:29.247 INFO:teuthology.orchestra.run.smithi171.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-05-22T10:27:29.247 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-05-22T10:27:29.247 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-05-22T10:27:29.248 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server 2023-05-22T10:27:29.369 INFO:teuthology.dispatcher.supervisor:Running job 7282307 2023-05-22T10:27:29.369 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_01abd1ed90624b09fd81d9cb3a79e2834351f0ca/virtualenv/bin/teuthology -v --owner scheduled_adking@teuthology --archive /home/teuthworker/archive/adking-2023-05-22_01:46:07-orch:cephadm-wip-adk2-testing-2023-05-21-1627-reef-distro-default-smithi/7282307 --name adking-2023-05-22_01:46:07-orch:cephadm-wip-adk2-testing-2023-05-21-1627-reef-distro-default-smithi --description orch:cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} -- /home/teuthworker/archive/adking-2023-05-22_01:46:07-orch:cephadm-wip-adk2-testing-2023-05-21-1627-reef-distro-default-smithi/7282307/orig.config.yaml 2023-05-22T10:27:29.373 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/adking-2023-05-22_01:46:07-orch:cephadm-wip-adk2-testing-2023-05-21-1627-reef-distro-default-smithi/7282307 2023-05-22T10:27:29.373 INFO:teuthology.dispatcher.supervisor:Job PID: 26991 2023-05-22T10:27:29.374 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-05-22T18:51:27.422 ERROR:teuthology.report:POST to https://paddles.front.sepia.ceph.com/runs/adking-2023-05-22_01:46:07-orch:cephadm-wip-adk2-testing-2023-05-21-1627-reef-distro-default-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2023-05-22T22:27:45.421 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-05-22T22:27:45.565 INFO:teuthology.kill:Killing Pids: {26991} 2023-05-22T22:27:46.378 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-05-22T22:27:46.379 INFO:teuthology.task.internal:roles: ubuntu@smithi171.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-05-22T22:27:46.379 INFO:teuthology.misc:Compressing logs... 2023-05-22T22:27:46.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-05-22T22:27:46.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2023-05-22T22:27:46.591 INFO:teuthology.orchestra.run.smithi150.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-05-22T22:28:01.610 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner 2023-05-22T22:28:01.613 ERROR:paramiko.transport:Traceback (most recent call last): 2023-05-22T22:28:01.613 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_01abd1ed90624b09fd81d9cb3a79e2834351f0ca/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner 2023-05-22T22:28:01.613 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2023-05-22T22:28:01.613 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_01abd1ed90624b09fd81d9cb3a79e2834351f0ca/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline 2023-05-22T22:28:01.614 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2023-05-22T22:28:01.614 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_01abd1ed90624b09fd81d9cb3a79e2834351f0ca/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 622, in _read_timeout 2023-05-22T22:28:01.614 ERROR:paramiko.transport: raise socket.timeout() 2023-05-22T22:28:01.614 ERROR:paramiko.transport:socket.timeout 2023-05-22T22:28:01.614 ERROR:paramiko.transport: 2023-05-22T22:28:01.615 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2023-05-22T22:28:01.615 ERROR:paramiko.transport: 2023-05-22T22:28:01.615 ERROR:paramiko.transport:Traceback (most recent call last): 2023-05-22T22:28:01.615 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_01abd1ed90624b09fd81d9cb3a79e2834351f0ca/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2093, in run 2023-05-22T22:28:01.615 ERROR:paramiko.transport: self._check_banner() 2023-05-22T22:28:01.616 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_01abd1ed90624b09fd81d9cb3a79e2834351f0ca/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2274, in _check_banner 2023-05-22T22:28:01.616 ERROR:paramiko.transport: raise SSHException( 2023-05-22T22:28:01.616 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2023-05-22T22:28:01.619 ERROR:paramiko.transport: 2023-05-22T22:28:01.620 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_01abd1ed90624b09fd81d9cb3a79e2834351f0ca/teuthology/dispatcher/supervisor.py", line 313, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_01abd1ed90624b09fd81d9cb3a79e2834351f0ca/teuthology/dispatcher/supervisor.py", line 378, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_01abd1ed90624b09fd81d9cb3a79e2834351f0ca/teuthology/misc.py", line 1368, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_01abd1ed90624b09fd81d9cb3a79e2834351f0ca/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_01abd1ed90624b09fd81d9cb3a79e2834351f0ca/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_01abd1ed90624b09fd81d9cb3a79e2834351f0ca/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi171 2023-05-22T22:28:01.668 INFO:teuthology.kill:No teuthology processes running 2023-05-22T22:28:02.415 INFO:teuthology.kill:Nuking machines: ['smithi150', 'smithi171'] 2023-05-22T22:28:02.976 INFO:teuthology.kill:2023-05-22 22:28:02,976.976 INFO:teuthology.nuke:targets: 2023-05-22T22:28:02.977 INFO:teuthology.kill: smithi150.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxw8VSYjmLjIt3sycpnENJ++fxzRKPYmNrXFJxyPQ/0N2MV6zLT6XoWGv3yyLEQgAwtjsz/DJq15McYVEt1nXG7/mnjwxrLrH+5RF96yoCds+xlPU9LbkAYhr5x3gLfR26BehsL6Z1TjhtKK6l3dmIkDx8xjcli3fE31uLhwSvT9Kwt0GZRmo9bRcOEipM3Fp9ReorPqgkJbLhI+Te+aR9FSYjFf1RpQq4CbBMFeo1afBEo2h+mHst3xe/rOa2B2JQNKreFfLXUPa3pPHaGxZPdhTYilRMRpUvWag0k89S4Zty+ZApaZGYjcnTKfKm9lACGUEjC8g0sALM4YgvQcCK6G+eCNpPUd8bAgfkZKZxsDJWSSRUTPtcSGDw218gDk9H4+4Omr4BVdNS0auyXNR2ppbREzdqZsIsQkBdSEazd/lrfd5+Bxmo2GYMgL6p4ulJrEG+oPJbFW8s8KL4vqSojgUzugm0yaFndUcDEHO7PPB8AX+XJKEWKzZudQ3Xx80= 2023-05-22T22:28:02.977 INFO:teuthology.kill: smithi171.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAqjQPfFhDYN7L8wP3HznVFlJXVfx1YsgmdJaDEiWislENbodYht2OA/w5pnayZ+chdRN9TbmoSTvdreAhABPrI= 2023-05-22T22:28:02.977 INFO:teuthology.kill:2023-05-22 22:28:02,976.976 INFO:teuthology.nuke:Checking targets against current locks 2023-05-22T22:28:03.033 INFO:teuthology.kill:2023-05-22 22:28:03,033.033 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-05-22T22:28:03.056 INFO:teuthology.kill:2023-05-22 22:28:03,056.056 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-05-22T22:28:03.107 INFO:teuthology.kill:2023-05-22 22:28:03,107.107 INFO:teuthology.orchestra.console:Power off smithi150 2023-05-22T22:28:03.120 INFO:teuthology.kill:2023-05-22 22:28:03,119.119 INFO:teuthology.orchestra.console:Power off smithi171 2023-05-22T22:28:11.475 INFO:teuthology.kill:2023-05-22 22:28:11,475.475 INFO:teuthology.orchestra.console:Power off for smithi150 completed 2023-05-22T22:28:11.647 INFO:teuthology.kill:2023-05-22 22:28:11,646.646 INFO:teuthology.lock.ops:unlocked: smithi150.front.sepia.ceph.com 2023-05-22T22:28:15.594 INFO:teuthology.kill:2023-05-22 22:28:15,594.594 INFO:teuthology.orchestra.console:Power off for smithi171 completed 2023-05-22T22:28:15.744 INFO:teuthology.kill:2023-05-22 22:28:15,744.744 INFO:teuthology.lock.ops:unlocked: smithi171.front.sepia.ceph.com 2023-05-22T22:30:16.052 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15