2023-04-27T18:41:21.674 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae
2023-04-27T18:41:21.823 INFO:teuthology.lock.ops:Start node 'smithi110.front.sepia.ceph.com' reimaging
2023-04-27T18:41:21.825 INFO:teuthology.lock.ops:Updating [smithi110.front.sepia.ceph.com]: reset os type and version on server
2023-04-27T18:41:21.825 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server
2023-04-27T18:41:21.847 INFO:teuthology.lock.ops:Node 'smithi110.front.sepia.ceph.com' reimaging is complete
2023-04-27T18:41:21.848 INFO:teuthology.lock.ops:Start node 'smithi148.front.sepia.ceph.com' reimaging
2023-04-27T18:41:21.848 INFO:teuthology.lock.ops:Updating [smithi148.front.sepia.ceph.com]: reset os type and version on server
2023-04-27T18:41:21.848 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server
2023-04-27T18:41:21.872 INFO:teuthology.lock.ops:Node 'smithi148.front.sepia.ceph.com' reimaging is complete
2023-04-27T18:41:22.017 INFO:teuthology.provision.fog.smithi110:Scheduling deploy of centos 8.stream
2023-04-27T18:41:22.036 INFO:teuthology.provision.fog.smithi148:Scheduling deploy of centos 8.stream
2023-04-27T18:41:22.235 INFO:teuthology.orchestra.console:Power off smithi148
2023-04-27T18:41:22.252 INFO:teuthology.orchestra.console:Power off smithi110
2023-04-27T18:41:22.389 INFO:teuthology.orchestra.console:Power off for smithi148 completed
2023-04-27T18:41:22.398 INFO:teuthology.orchestra.console:Power off for smithi110 completed
2023-04-27T18:41:22.490 INFO:teuthology.orchestra.console:Power on smithi148
2023-04-27T18:41:22.502 INFO:teuthology.orchestra.console:Power on smithi110
2023-04-27T18:41:26.763 INFO:teuthology.orchestra.console:Power on for smithi110 completed
2023-04-27T18:41:26.864 INFO:teuthology.provision.fog.smithi110:Waiting for deploy to finish
2023-04-27T18:41:30.869 INFO:teuthology.orchestra.console:Power on for smithi148 completed
2023-04-27T18:41:30.970 INFO:teuthology.provision.fog.smithi148:Waiting for deploy to finish
2023-04-27T18:44:31.109 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.110
2023-04-27T18:44:34.949 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.148
2023-04-27T18:44:40.169 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.110
2023-04-27T18:44:44.005 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.148
2023-04-27T18:44:49.253 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.110
2023-04-27T18:44:53.061 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.148
2023-04-27T18:44:58.309 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.110
2023-04-27T18:45:02.117 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.148
2023-04-27T18:45:07.369 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.110
2023-04-27T18:45:11.173 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.148
2023-04-27T18:45:20.229 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.148
2023-04-27T18:45:57.869 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-04-27T18:46:13.374 WARNING:teuthology.provision.fog:timed out
2023-04-27T18:46:19.773 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-04-27T18:46:20.047 INFO:teuthology.provision.fog.smithi110:Node is ready
2023-04-27T18:46:20.106 INFO:teuthology.orchestra.run.smithi110.stdout:smithi110.front.sepia.ceph.com
2023-04-27T18:46:20.164 INFO:teuthology.orchestra.run.smithi110.stdout:172.21.15.110 smithi110.front.sepia.ceph.com smithi110
2023-04-27T18:46:20.526 INFO:teuthology.provision.fog.smithi110:Deploy complete!
2023-04-27T18:46:22.916 INFO:teuthology.provision.fog.smithi148:Node is ready
2023-04-27T18:46:22.939 INFO:teuthology.orchestra.run.smithi148.stdout:smithi148.front.sepia.ceph.com
2023-04-27T18:46:22.998 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148 smithi148.front.sepia.ceph.com smithi148
2023-04-27T18:46:23.478 INFO:teuthology.provision.fog.smithi148:Deploy complete!
2023-04-27T18:46:23.653 INFO:teuthology.lock.ops:Checking smithi110.front.sepia.ceph.com
2023-04-27T18:46:23.654 INFO:teuthology.lock.ops:New key found. Updating...
2023-04-27T18:46:23.686 INFO:teuthology.lock.ops:Checking smithi148.front.sepia.ceph.com
2023-04-27T18:46:23.686 INFO:teuthology.lock.ops:New key found. Updating...
2023-04-27T18:46:23.712 INFO:teuthology.lock.ops:Updating [smithi110.front.sepia.ceph.com]: set os type and version on server
2023-04-27T18:46:23.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com'
2023-04-27T18:46:23.906 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64
2023-04-27T18:46:23.966 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="CentOS Stream"
2023-04-27T18:46:23.966 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="8"
2023-04-27T18:46:23.966 INFO:teuthology.orchestra.run.smithi110.stdout:ID="centos"
2023-04-27T18:46:23.967 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="rhel fedora"
2023-04-27T18:46:23.967 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="8"
2023-04-27T18:46:23.967 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el8"
2023-04-27T18:46:23.967 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="CentOS Stream 8"
2023-04-27T18:46:23.967 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31"
2023-04-27T18:46:23.967 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-04-27T18:46:23.967 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://centos.org/"
2023-04-27T18:46:23.968 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-04-27T18:46:23.968 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-04-27T18:46:23.968 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-04-27T18:46:23.969 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server
2023-04-27T18:46:23.994 INFO:teuthology.lock.ops:Updating [smithi148.front.sepia.ceph.com]: set os type and version on server
2023-04-27T18:46:23.994 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com'
2023-04-27T18:46:24.185 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64
2023-04-27T18:46:24.242 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Stream"
2023-04-27T18:46:24.242 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="8"
2023-04-27T18:46:24.242 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos"
2023-04-27T18:46:24.242 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora"
2023-04-27T18:46:24.243 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="8"
2023-04-27T18:46:24.243 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el8"
2023-04-27T18:46:24.243 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Stream 8"
2023-04-27T18:46:24.243 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31"
2023-04-27T18:46:24.243 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-04-27T18:46:24.243 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://centos.org/"
2023-04-27T18:46:24.243 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-04-27T18:46:24.244 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-04-27T18:46:24.244 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-04-27T18:46:24.244 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server
2023-04-27T18:46:24.378 INFO:teuthology.dispatcher.supervisor:Running job 7251092
2023-04-27T18:46:24.378 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/bin/teuthology -v --owner scheduled_gabrioux@teuthology --archive /home/teuthworker/archive/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251092 --name gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251092/orig.config.yaml
2023-04-27T18:46:24.383 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/gabrioux-2023-04-25_13:08:27-orch:cephadm-wip-guits-testing-2023-04-25-0823-distro-default-smithi/7251092
2023-04-27T18:46:24.384 INFO:teuthology.dispatcher.supervisor:Job PID: 9678
2023-04-27T18:46:24.384 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-04-28T06:47:21.481 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-04-28T06:47:21.654 INFO:teuthology.kill:Killing Pids: {9678}
2023-04-28T06:47:31.089 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2023-04-28T06:47:31.089 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2023-04-28T06:47:31.090 INFO:teuthology.misc:Compressing logs...
2023-04-28T06:47:31.090 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com'
2023-04-28T06:47:31.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com'
2023-04-28T06:47:46.365 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner
2023-04-28T06:47:46.463 ERROR:paramiko.transport:Traceback (most recent call last):
2023-04-28T06:47:46.463 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-04-28T06:47:46.463 ERROR:paramiko.transport:    buf = self.packetizer.readline(timeout)
2023-04-28T06:47:46.464 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline
2023-04-28T06:47:46.464 ERROR:paramiko.transport:    buf += self._read_timeout(timeout)
2023-04-28T06:47:46.464 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 622, in _read_timeout
2023-04-28T06:47:46.464 ERROR:paramiko.transport:    raise socket.timeout()
2023-04-28T06:47:46.465 ERROR:paramiko.transport:socket.timeout
2023-04-28T06:47:46.465 ERROR:paramiko.transport:
2023-04-28T06:47:46.465 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2023-04-28T06:47:46.465 ERROR:paramiko.transport:
2023-04-28T06:47:46.466 ERROR:paramiko.transport:Traceback (most recent call last):
2023-04-28T06:47:46.466 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2093, in run
2023-04-28T06:47:46.466 ERROR:paramiko.transport:    self._check_banner()
2023-04-28T06:47:46.467 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2274, in _check_banner
2023-04-28T06:47:46.467 ERROR:paramiko.transport:    raise SSHException(
2023-04-28T06:47:46.467 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-04-28T06:47:46.467 ERROR:paramiko.transport:
2023-04-28T06:47:46.468 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/dispatcher/supervisor.py", line 313, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/dispatcher/supervisor.py", line 378, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/misc.py", line 1368, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_8d156aede5efdae00b53d8d3b8d127082980e7ec/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi148
2023-04-28T06:47:46.534 INFO:teuthology.kill:No teuthology processes running
2023-04-28T06:47:47.400 INFO:teuthology.kill:Nuking machines: ['smithi110', 'smithi148']
2023-04-28T06:47:48.066 INFO:teuthology.kill:2023-04-28 06:47:48,066.066 INFO:teuthology.nuke:targets:
2023-04-28T06:47:48.067 INFO:teuthology.kill:  smithi110.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD3lJA11seSQo/t8Kz0yhZEP4DA3zXBVfJF9pJDOLfa36Fh3trpcoOA51FcFHo+6iTyAHChiuoPZvlN9w0vNdTCNqOPOBNC8fa+BxlFbJzr9T5nXm4q7LJLs/qTRM3yrgq6DEbdim6vVnD3+7O2sW1pbkR5EYa9EcBks0B91NixPUtGbKhxl7Xtrf6eVxVpnQoBtL3GaX/pZsJfgaJixZoxz+0+wQfHt6A25GQrBCMqVQXeNNsnSc7xoVpi7+KxSbg16EdSDiTB/B8fBkYh4YlGu+04FtkaUxJrxQE/W58aHd6sZy1chpNTIcBc0EiEmvVVc7s+0mCLQx4PwY0l4tlCwVSSe6WsaRP2GqbPC1IUUWSiS7ZIwRv4kjdn0/sFtG10b7GTxR2p9WBBXfamf0X3cwhdRJrO+ErVOPhqvAc9bb982lmKehdo6jCF8IPlrG9FP0uO9X6qo1qacnBN/4qX4LZwEGMIGx8mv7Q9Dk4yRo5TJaMLWAPBgn31Pgdi5Cc=
2023-04-28T06:47:48.067 INFO:teuthology.kill:  smithi148.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEyHYQVd3y9NBFWmq67XflAR7Dv1w9XmbOXBEwfuShVXLyH8IEq7Dopa7+z9umgamE2WVnP9yICPPnHFSFhENUKBF0YHJv/+cN46Z2bp7Rs5Oxm2zrDoEepgHGxq3kpMUUvy/9MvM1/y9p0QHw+VDIY2BQukUrJrp5AWYFDbai12FENg6dCdGP8wVueZ1MIrd9RJObPbc+gySQ1WTsNwDotVXcJWfnG0RYcnsKQsYOfDmPEhvOE4KBBY++XJJ+v6JI7XGWfAVOTPpcv+3IDopPKiC3mlol0TYVSym6YLL2iTuMMtQMEhOjzutZFI0oZClJ9slzPfreHS2yvkzAUobPIXNipOE4z4ePD0S0F8jQnII1OOKCeezF99aIf2cY9nxW23c41QpDTvt853KztDXDPmfrGey6i5AYQPgrbGeX2KLNLTCmz95bM1/vMp0MiqF7+oqhasD3Yw59Omnn5cT8RSO8Txqrxt2rkYGUuVatPe+7R68j2EPC3dB14BSiIWk=
2023-04-28T06:47:48.068 INFO:teuthology.kill:2023-04-28 06:47:48,066.066 INFO:teuthology.nuke:Checking targets against current locks
2023-04-28T06:47:48.121 INFO:teuthology.kill:2023-04-28 06:47:48,121.121 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-04-28T06:47:48.150 INFO:teuthology.kill:2023-04-28 06:47:48,150.150 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-04-28T06:47:48.204 INFO:teuthology.kill:2023-04-28 06:47:48,204.204 INFO:teuthology.orchestra.console:Power off smithi110
2023-04-28T06:47:48.241 INFO:teuthology.kill:2023-04-28 06:47:48,240.240 INFO:teuthology.orchestra.console:Power off smithi148
2023-04-28T06:48:00.687 INFO:teuthology.kill:2023-04-28 06:48:00,687.687 INFO:teuthology.orchestra.console:Power off for smithi110 completed
2023-04-28T06:48:00.719 INFO:teuthology.kill:2023-04-28 06:48:00,719.719 INFO:teuthology.orchestra.console:Power off for smithi148 completed
2023-04-28T06:48:00.849 INFO:teuthology.kill:2023-04-28 06:48:00,849.849 INFO:teuthology.lock.ops:unlocked: smithi110.front.sepia.ceph.com
2023-04-28T06:48:00.872 INFO:teuthology.kill:2023-04-28 06:48:00,872.872 INFO:teuthology.lock.ops:unlocked: smithi148.front.sepia.ceph.com
2023-04-28T06:50:01.221 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-04-28T06:50:01.341 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi110 but it was locked by another job: /home/teuthworker/archive/sseshasa-2023-04-28_06:06:11-rados-wip-sseshasa3-testing-2023-04-27-1947-distro-default-smithi/7256371
2023-04-28T06:50:01.342 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi148 but it was locked by another job: /home/teuthworker/archive/sseshasa-2023-04-28_06:06:11-rados-wip-sseshasa3-testing-2023-04-27-1947-distro-default-smithi/7256371