2023-05-06T19:35:47.429 INFO:root:teuthology version: 0.0.1.dev98+g69caab8
2023-05-06T19:35:47.577 INFO:teuthology.lock.ops:Start node 'smithi151.front.sepia.ceph.com' reimaging
2023-05-06T19:35:47.578 INFO:teuthology.lock.ops:Updating [smithi151.front.sepia.ceph.com]: reset os type and version on server
2023-05-06T19:35:47.578 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server
2023-05-06T19:35:47.604 INFO:teuthology.lock.ops:Node 'smithi151.front.sepia.ceph.com' reimaging is complete
2023-05-06T19:35:47.604 INFO:teuthology.lock.ops:Start node 'smithi202.front.sepia.ceph.com' reimaging
2023-05-06T19:35:47.604 INFO:teuthology.lock.ops:Updating [smithi202.front.sepia.ceph.com]: reset os type and version on server
2023-05-06T19:35:47.604 INFO:teuthology.lock.ops:Updating smithi202.front.sepia.ceph.com on lock server
2023-05-06T19:35:47.628 INFO:teuthology.lock.ops:Node 'smithi202.front.sepia.ceph.com' reimaging is complete
2023-05-06T19:35:47.763 INFO:teuthology.provision.fog.smithi151:Scheduling deploy of centos 8.stream
2023-05-06T19:35:47.772 INFO:teuthology.provision.fog.smithi202:Scheduling deploy of centos 8.stream
2023-05-06T19:35:48.039 INFO:teuthology.orchestra.console:Power off smithi151
2023-05-06T19:35:48.056 INFO:teuthology.orchestra.console:Power off smithi202
2023-05-06T19:36:00.527 INFO:teuthology.orchestra.console:Power off for smithi151 completed
2023-05-06T19:36:00.534 INFO:teuthology.orchestra.console:Power off for smithi202 completed
2023-05-06T19:36:00.629 INFO:teuthology.orchestra.console:Power on smithi151
2023-05-06T19:36:00.638 INFO:teuthology.orchestra.console:Power on smithi202
2023-05-06T19:36:04.886 INFO:teuthology.orchestra.console:Power on for smithi151 completed
2023-05-06T19:36:04.894 INFO:teuthology.orchestra.console:Power on for smithi202 completed
2023-05-06T19:36:04.988 INFO:teuthology.provision.fog.smithi151:Waiting for deploy to finish
2023-05-06T19:36:04.995 INFO:teuthology.provision.fog.smithi202:Waiting for deploy to finish
2023-05-06T19:38:54.100 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151
2023-05-06T19:38:54.101 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.202
2023-05-06T19:39:03.184 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151
2023-05-06T19:39:03.185 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.202
2023-05-06T19:39:12.240 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151
2023-05-06T19:39:12.241 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.202
2023-05-06T19:39:21.300 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151
2023-05-06T19:39:21.301 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.202
2023-05-06T19:39:30.352 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151
2023-05-06T19:39:30.353 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.202
2023-05-06T19:39:39.440 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151
2023-05-06T19:39:48.496 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151
2023-05-06T19:39:57.584 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151
2023-05-06T19:40:35.278 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-06T19:40:36.362 WARNING:teuthology.provision.fog:timed out
2023-05-06T19:40:42.571 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-05-06T19:40:42.839 INFO:teuthology.provision.fog.smithi202:Node is ready
2023-05-06T19:40:42.860 INFO:teuthology.orchestra.run.smithi202.stdout:smithi202.front.sepia.ceph.com
2023-05-06T19:40:42.922 INFO:teuthology.orchestra.run.smithi202.stdout:172.21.15.202 smithi202.front.sepia.ceph.com smithi202
2023-05-06T19:40:43.292 INFO:teuthology.provision.fog.smithi202:Deploy complete!
2023-05-06T19:41:00.993 INFO:teuthology.provision.fog.smithi151:Node is ready
2023-05-06T19:41:01.018 INFO:teuthology.orchestra.run.smithi151.stdout:smithi151.front.sepia.ceph.com
2023-05-06T19:41:01.080 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151 smithi151.front.sepia.ceph.com smithi151
2023-05-06T19:41:01.450 INFO:teuthology.provision.fog.smithi151:Deploy complete!
2023-05-06T19:41:01.652 INFO:teuthology.lock.ops:Checking smithi151.front.sepia.ceph.com
2023-05-06T19:41:01.653 INFO:teuthology.lock.ops:New key found. Updating...
2023-05-06T19:41:01.674 INFO:teuthology.lock.ops:Checking smithi202.front.sepia.ceph.com
2023-05-06T19:41:01.674 INFO:teuthology.lock.ops:New key found. Updating...
2023-05-06T19:41:01.700 INFO:teuthology.lock.ops:Updating [smithi151.front.sepia.ceph.com]: set os type and version on server
2023-05-06T19:41:01.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com'
2023-05-06T19:41:01.896 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64
2023-05-06T19:41:01.955 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="CentOS Stream"
2023-05-06T19:41:01.955 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="8"
2023-05-06T19:41:01.956 INFO:teuthology.orchestra.run.smithi151.stdout:ID="centos"
2023-05-06T19:41:01.956 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE="rhel fedora"
2023-05-06T19:41:01.956 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="8"
2023-05-06T19:41:01.956 INFO:teuthology.orchestra.run.smithi151.stdout:PLATFORM_ID="platform:el8"
2023-05-06T19:41:01.956 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="CentOS Stream 8"
2023-05-06T19:41:01.956 INFO:teuthology.orchestra.run.smithi151.stdout:ANSI_COLOR="0;31"
2023-05-06T19:41:01.957 INFO:teuthology.orchestra.run.smithi151.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-05-06T19:41:01.957 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://centos.org/"
2023-05-06T19:41:01.957 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-05-06T19:41:01.957 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-05-06T19:41:01.957 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-05-06T19:41:01.958 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server
2023-05-06T19:41:01.983 INFO:teuthology.lock.ops:Updating [smithi202.front.sepia.ceph.com]: set os type and version on server
2023-05-06T19:41:01.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi202.front.sepia.ceph.com'
2023-05-06T19:41:02.247 INFO:teuthology.orchestra.run.smithi202.stdout:x86_64
2023-05-06T19:41:02.306 INFO:teuthology.orchestra.run.smithi202.stdout:NAME="CentOS Stream"
2023-05-06T19:41:02.306 INFO:teuthology.orchestra.run.smithi202.stdout:VERSION="8"
2023-05-06T19:41:02.306 INFO:teuthology.orchestra.run.smithi202.stdout:ID="centos"
2023-05-06T19:41:02.306 INFO:teuthology.orchestra.run.smithi202.stdout:ID_LIKE="rhel fedora"
2023-05-06T19:41:02.306 INFO:teuthology.orchestra.run.smithi202.stdout:VERSION_ID="8"
2023-05-06T19:41:02.307 INFO:teuthology.orchestra.run.smithi202.stdout:PLATFORM_ID="platform:el8"
2023-05-06T19:41:02.307 INFO:teuthology.orchestra.run.smithi202.stdout:PRETTY_NAME="CentOS Stream 8"
2023-05-06T19:41:02.307 INFO:teuthology.orchestra.run.smithi202.stdout:ANSI_COLOR="0;31"
2023-05-06T19:41:02.307 INFO:teuthology.orchestra.run.smithi202.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-05-06T19:41:02.307 INFO:teuthology.orchestra.run.smithi202.stdout:HOME_URL="https://centos.org/"
2023-05-06T19:41:02.307 INFO:teuthology.orchestra.run.smithi202.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-05-06T19:41:02.307 INFO:teuthology.orchestra.run.smithi202.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-05-06T19:41:02.308 INFO:teuthology.orchestra.run.smithi202.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-05-06T19:41:02.308 INFO:teuthology.lock.ops:Updating smithi202.front.sepia.ceph.com on lock server
2023-05-06T19:41:02.425 INFO:teuthology.dispatcher.supervisor:Running job 7264403
2023-05-06T19:41:02.425 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-05-06_14:41:44-rados-pacific-release-distro-default-smithi/7264403 --name yuriw-2023-05-06_14:41:44-rados-pacific-release-distro-default-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2023-05-06_14:41:44-rados-pacific-release-distro-default-smithi/7264403/orig.config.yaml
2023-05-06T19:41:02.429 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-05-06_14:41:44-rados-pacific-release-distro-default-smithi/7264403
2023-05-06T19:41:02.430 INFO:teuthology.dispatcher.supervisor:Job PID: 27190
2023-05-06T19:41:02.430 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-05-07T07:42:00.584 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-05-07T07:42:00.746 INFO:teuthology.kill:Killing Pids: {27190}
2023-05-07T07:42:01.848 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2023-05-07T07:42:01.849 INFO:teuthology.task.internal:roles: ubuntu@smithi202.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2023-05-07T07:42:01.850 INFO:teuthology.misc:Compressing logs...
2023-05-07T07:42:01.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com'
2023-05-07T07:42:02.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi202.front.sepia.ceph.com'
2023-05-07T07:42:17.192 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner
2023-05-07T07:42:17.232 ERROR:paramiko.transport:Traceback (most recent call last):
2023-05-07T07:42:17.232 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-05-07T07:42:17.232 ERROR:paramiko.transport:    buf = self.packetizer.readline(timeout)
2023-05-07T07:42:17.233 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline
2023-05-07T07:42:17.233 ERROR:paramiko.transport:    buf += self._read_timeout(timeout)
2023-05-07T07:42:17.233 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 622, in _read_timeout
2023-05-07T07:42:17.233 ERROR:paramiko.transport:    raise socket.timeout()
2023-05-07T07:42:17.233 ERROR:paramiko.transport:socket.timeout
2023-05-07T07:42:17.234 ERROR:paramiko.transport:
2023-05-07T07:42:17.234 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2023-05-07T07:42:17.234 ERROR:paramiko.transport:
2023-05-07T07:42:17.234 ERROR:paramiko.transport:Traceback (most recent call last):
2023-05-07T07:42:17.234 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2093, in run
2023-05-07T07:42:17.235 ERROR:paramiko.transport:    self._check_banner()
2023-05-07T07:42:17.235 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2274, in _check_banner
2023-05-07T07:42:17.235 ERROR:paramiko.transport:    raise SSHException(
2023-05-07T07:42:17.235 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-05-07T07:42:17.236 ERROR:paramiko.transport:
2023-05-07T07:42:17.236 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/dispatcher/supervisor.py", line 313, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/dispatcher/supervisor.py", line 378, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/misc.py", line 1368, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_69caab8ec6f44c61c991ae693e8e7b8cd859a896/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi202
2023-05-07T07:42:17.328 INFO:teuthology.kill:No teuthology processes running
2023-05-07T07:42:18.268 INFO:teuthology.kill:Nuking machines: ['smithi151', 'smithi202']
2023-05-07T07:42:19.017 INFO:teuthology.kill:2023-05-07 07:42:19,016.016 INFO:teuthology.nuke:targets:
2023-05-07T07:42:19.018 INFO:teuthology.kill:  smithi151.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUEbtznbrrwLzTXIwgdQA9Dzdb4iqKqOLZvMQ5DjwvIBWCi3MG9VuYpUvGMqMNzE+5djd+IER3h64p14aApkQqQtOtLwC3lnz6Ky3at4rAx4pfOJzTe2S2XPLaKiA+Sd2NCiZ7uAr1MaL11CDRzrX/i64NLss/DDuY3SO++H/QzqYI4lDCjv9UtbUcEAS4CmORc5YXIeU92lOw2gWSLX7nz102lblP/ekq+LyH3M6jeH7XD6QDMScje+iBxiaFWVDTcgCwQyiLCR6fnQvuyGR+C7SwYBnGWgLdv9GwdqkeIhzzP5bcJcNdghruwZQzdNqKJ3GdoGbS1334EU3wB5javrFf9Wyq1bzN0c0V2tMieG7Wlh3dQGk5TkRvMO9fakCy4TIDK2oM5dHJZb5lDEYMFIzIkw2ktOIhvP/720mtnpawxDOkSk2kx6P3CTTv0Z3mCHDnK52nYLy3clB50Yk/xLCRJ1IZ+6mpQLOPEfJHXHR/cg3clmZgcANpYJHiiMk=
2023-05-07T07:42:19.018 INFO:teuthology.kill:  smithi202.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlW0LVJ9xqj+Nn2ac/Ooxh5nlcLgH5E0A+CCnlUZrptEqV14obu/kZnJHlLatJjLXRcUqdV/I436GcwmsUVNi9GWTn85aUdVKeB/iwFCjZYWsa4/vIcJhSJcdpgMBDYBoFcX9gE/QjQgq0F1ycibDqbK9L5QPay2nHpQviMRjU/u95wVOy71mB+tEMRwy9X48jiu9Vn63fUbK6z8Xt6VX/Hi9G/Mc8yHci7KdgnxGMZCELlVMrZj7R4kReyc+K1fyEWqsuDr9t6XeVBBj3mB0NaTr800zUFmK0uH1bY5jjuAlPTna47s9kqZNlq2YeOqXC2Kp4+4c+m3/bfVAs9823aQJ/Ce7LLPXKmD/nWxH1zybYNQPsvivc9w2b7zlXh8M4PCNDm8CVUHHi0DFmtpC6ByfXXfEFcl57tI8wzhs4MJKZlhSwUhlHv7Mxnn1Xivx6sQXaNaG5DYNSKux9s2N4yxn1ATfviAI2UCo1B01CQb5ypgrKDclayDwAcgn1ja0=
2023-05-07T07:42:19.018 INFO:teuthology.kill:2023-05-07 07:42:19,017.017 INFO:teuthology.nuke:Checking targets against current locks
2023-05-07T07:42:19.073 INFO:teuthology.kill:2023-05-07 07:42:19,073.073 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-07T07:42:19.101 INFO:teuthology.kill:2023-05-07 07:42:19,101.101 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-05-07T07:42:19.163 INFO:teuthology.kill:2023-05-07 07:42:19,162.162 INFO:teuthology.orchestra.console:Power off smithi151
2023-05-07T07:42:19.203 INFO:teuthology.kill:2023-05-07 07:42:19,202.202 INFO:teuthology.orchestra.console:Power off smithi202
2023-05-07T07:42:27.572 INFO:teuthology.kill:2023-05-07 07:42:27,572.572 INFO:teuthology.orchestra.console:Power off for smithi202 completed
2023-05-07T07:42:27.725 INFO:teuthology.kill:2023-05-07 07:42:27,725.725 INFO:teuthology.lock.ops:unlocked: smithi202.front.sepia.ceph.com
2023-05-07T07:42:31.659 INFO:teuthology.kill:2023-05-07 07:42:31,658.658 INFO:teuthology.orchestra.console:Power off for smithi151 completed
2023-05-07T07:42:31.818 INFO:teuthology.kill:2023-05-07 07:42:31,818.818 INFO:teuthology.lock.ops:unlocked: smithi151.front.sepia.ceph.com
2023-05-07T07:44:32.207 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-05-07T07:44:32.325 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi151 but it was locked by another job: /home/teuthworker/archive/yuriw-2023-05-06_14:40:51-fs-reef-distro-default-smithi/7263883