2022-07-03T14:57:07.101 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3
2022-07-03T14:57:07.251 INFO:teuthology.lock.ops:Start node 'smithi019.front.sepia.ceph.com' reimaging
2022-07-03T14:57:07.252 INFO:teuthology.lock.ops:Updating [smithi019.front.sepia.ceph.com]: reset os type and version on server
2022-07-03T14:57:07.252 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server
2022-07-03T14:57:07.274 INFO:teuthology.lock.ops:Node 'smithi019.front.sepia.ceph.com' reimaging is complete
2022-07-03T14:57:07.275 INFO:teuthology.lock.ops:Start node 'smithi059.front.sepia.ceph.com' reimaging
2022-07-03T14:57:07.275 INFO:teuthology.lock.ops:Updating [smithi059.front.sepia.ceph.com]: reset os type and version on server
2022-07-03T14:57:07.276 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server
2022-07-03T14:57:07.299 INFO:teuthology.lock.ops:Node 'smithi059.front.sepia.ceph.com' reimaging is complete
2022-07-03T14:57:07.486 INFO:teuthology.provision.fog.smithi019:Scheduling deploy of centos 8.stream
2022-07-03T14:57:07.510 INFO:teuthology.provision.fog.smithi059:Scheduling deploy of centos 8.stream
2022-07-03T14:57:07.842 INFO:teuthology.orchestra.console:Power off smithi019
2022-07-03T14:57:07.863 INFO:teuthology.orchestra.console:Power off smithi059
2022-07-03T14:57:07.995 INFO:teuthology.orchestra.console:Power off for smithi019 completed
2022-07-03T14:57:08.003 INFO:teuthology.orchestra.console:Power off for smithi059 completed
2022-07-03T14:57:08.097 INFO:teuthology.orchestra.console:Power on smithi019
2022-07-03T14:57:08.105 INFO:teuthology.orchestra.console:Power on smithi059
2022-07-03T14:57:16.464 INFO:teuthology.orchestra.console:Power on for smithi019 completed
2022-07-03T14:57:16.467 INFO:teuthology.orchestra.console:Power on for smithi059 completed
2022-07-03T14:57:16.566 INFO:teuthology.provision.fog.smithi019:Waiting for deploy to finish
2022-07-03T14:57:16.569 INFO:teuthology.provision.fog.smithi059:Waiting for deploy to finish
2022-07-03T15:01:30.820 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-07-03T15:01:30.891 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-07-03T15:01:31.032 INFO:teuthology.provision.fog.smithi019:Node is ready
2022-07-03T15:01:31.052 INFO:teuthology.orchestra.run.smithi019.stdout:smithi019.front.sepia.ceph.com
2022-07-03T15:01:31.113 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19 smithi019.front.sepia.ceph.com smithi019
2022-07-03T15:01:31.176 INFO:teuthology.provision.fog.smithi059:Node is ready
2022-07-03T15:01:31.196 INFO:teuthology.orchestra.run.smithi059.stdout:smithi059.front.sepia.ceph.com
2022-07-03T15:01:31.266 INFO:teuthology.orchestra.run.smithi059.stdout:172.21.15.59 smithi059.front.sepia.ceph.com smithi059
2022-07-03T15:01:31.482 INFO:teuthology.provision.fog.smithi019:Deploy complete!
2022-07-03T15:01:31.638 INFO:teuthology.provision.fog.smithi059:Deploy complete!
2022-07-03T15:01:31.842 INFO:teuthology.lock.ops:Checking smithi019.front.sepia.ceph.com
2022-07-03T15:01:31.843 INFO:teuthology.lock.ops:New key found. Updating...
2022-07-03T15:01:31.866 INFO:teuthology.lock.ops:Checking smithi059.front.sepia.ceph.com
2022-07-03T15:01:31.866 INFO:teuthology.lock.ops:New key found. Updating...
2022-07-03T15:01:31.889 INFO:teuthology.lock.ops:Updating [smithi019.front.sepia.ceph.com]: set os type and version on server
2022-07-03T15:01:31.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2022-07-03T15:01:32.106 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64
2022-07-03T15:01:32.168 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="CentOS Stream"
2022-07-03T15:01:32.169 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="8"
2022-07-03T15:01:32.169 INFO:teuthology.orchestra.run.smithi019.stdout:ID="centos"
2022-07-03T15:01:32.170 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="rhel fedora"
2022-07-03T15:01:32.170 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="8"
2022-07-03T15:01:32.171 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el8"
2022-07-03T15:01:32.172 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="CentOS Stream 8"
2022-07-03T15:01:32.172 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31"
2022-07-03T15:01:32.173 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-07-03T15:01:32.173 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://centos.org/"
2022-07-03T15:01:32.173 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-07-03T15:01:32.174 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-07-03T15:01:32.174 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-07-03T15:01:32.177 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server
2022-07-03T15:01:32.214 INFO:teuthology.lock.ops:Updating [smithi059.front.sepia.ceph.com]: set os type and version on server
2022-07-03T15:01:32.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com'
2022-07-03T15:01:32.453 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64
2022-07-03T15:01:32.515 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="CentOS Stream"
2022-07-03T15:01:32.516 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="8"
2022-07-03T15:01:32.516 INFO:teuthology.orchestra.run.smithi059.stdout:ID="centos"
2022-07-03T15:01:32.516 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE="rhel fedora"
2022-07-03T15:01:32.517 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="8"
2022-07-03T15:01:32.517 INFO:teuthology.orchestra.run.smithi059.stdout:PLATFORM_ID="platform:el8"
2022-07-03T15:01:32.517 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="CentOS Stream 8"
2022-07-03T15:01:32.517 INFO:teuthology.orchestra.run.smithi059.stdout:ANSI_COLOR="0;31"
2022-07-03T15:01:32.518 INFO:teuthology.orchestra.run.smithi059.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-07-03T15:01:32.518 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://centos.org/"
2022-07-03T15:01:32.518 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-07-03T15:01:32.518 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-07-03T15:01:32.519 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-07-03T15:01:32.519 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server
2022-07-03T15:01:32.656 INFO:teuthology.dispatcher.supervisor:Running job 6911816
2022-07-03T15:01:32.657 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/bin/teuthology -v --owner scheduled_nshelke@teuthology --archive /home/teuthworker/archive/nshelke-2022-07-03_13:44:58-fs:volumes-wip-nshelke-in-prog-clones-02-07-2022-distro-default-smithi/6911816 --name nshelke-2022-07-03_13:44:58-fs:volumes-wip-nshelke-in-prog-clones-02-07-2022-distro-default-smithi --description fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} -- /home/teuthworker/archive/nshelke-2022-07-03_13:44:58-fs:volumes-wip-nshelke-in-prog-clones-02-07-2022-distro-default-smithi/6911816/orig.config.yaml
2022-07-03T15:01:32.661 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/nshelke-2022-07-03_13:44:58-fs:volumes-wip-nshelke-in-prog-clones-02-07-2022-distro-default-smithi/6911816
2022-07-03T15:01:32.662 INFO:teuthology.dispatcher.supervisor:Job PID: 2288
2022-07-03T15:01:32.663 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-07-04T03:02:29.147 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2022-07-04T03:02:29.322 INFO:teuthology.kill:Killing Pids: {2288}
2022-07-04T03:02:30.858 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2']
2022-07-04T03:02:30.859 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']
2022-07-04T03:02:30.859 INFO:teuthology.misc:Compressing logs...
2022-07-04T03:02:30.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2022-07-04T03:02:31.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com'
2022-07-04T03:03:31.344 ERROR:teuthology.orchestra.connection:Error authenticating with smithi059.front.sepia.ceph.com: Authentication timeout.
2022-07-04T03:03:32.742 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner[Errno 104] Connection reset by peer
2022-07-04T03:03:32.745 ERROR:paramiko.transport:Traceback (most recent call last):
2022-07-04T03:03:32.745 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2271, in _check_banner
2022-07-04T03:03:32.745 ERROR:paramiko.transport:    buf = self.packetizer.readline(timeout)
2022-07-04T03:03:32.746 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline
2022-07-04T03:03:32.746 ERROR:paramiko.transport:    buf += self._read_timeout(timeout)
2022-07-04T03:03:32.746 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 607, in _read_timeout
2022-07-04T03:03:32.747 ERROR:paramiko.transport:    x = self.__socket.recv(128)
2022-07-04T03:03:32.747 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/gevent/_socketcommon.py", line 657, in recv
2022-07-04T03:03:32.747 ERROR:paramiko.transport:    return self._sock.recv(*args)
2022-07-04T03:03:32.748 ERROR:paramiko.transport:ConnectionResetError: [Errno 104] Connection reset by peer
2022-07-04T03:03:32.748 ERROR:paramiko.transport:
2022-07-04T03:03:32.749 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2022-07-04T03:03:32.749 ERROR:paramiko.transport:
2022-07-04T03:03:32.750 ERROR:paramiko.transport:Traceback (most recent call last):
2022-07-04T03:03:32.750 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2094, in run
2022-07-04T03:03:32.751 ERROR:paramiko.transport:    self._check_banner()
2022-07-04T03:03:32.751 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2276, in _check_banner
2022-07-04T03:03:32.752 ERROR:paramiko.transport:    "Error reading SSH protocol banner" + str(e)
2022-07-04T03:03:32.752 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner[Errno 104] Connection reset by peer
2022-07-04T03:03:32.753 ERROR:paramiko.transport:
2022-07-04T03:03:32.755 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/dispatcher/supervisor.py", line 290, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/remote.py", line 509, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi059
2022-07-04T03:03:32.860 INFO:teuthology.kill:No teuthology processes running
2022-07-04T03:03:34.484 INFO:teuthology.kill:Nuking machines: ['smithi019', 'smithi059']
2022-07-04T03:03:36.006 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
2022-07-04T03:03:36.007 INFO:teuthology.kill:  from cryptography.hazmat.backends import default_backend
2022-07-04T03:03:36.008 INFO:teuthology.kill:2022-07-04 03:03:36,005.005 INFO:teuthology.nuke:targets:
2022-07-04T03:03:36.008 INFO:teuthology.kill:  smithi019.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1AFkn/VfkZLxnSk8z8whrEKHzrlVmLQSwCO4jb3VDA0JeyR3UWVp6ijJo34btzy9v4EI/gtVx7C5cBiHbfk2kAA2n2PTbwCAwX72hjnolCCiij+XpbBvhb7yM+7Mxc8DQHfNUgo2Vtiqvt992Tr/qJg9qVDAO7+iEKlqN1Dconl3HIMZ3t0F3ddkXt0z/Blo7xjElFYSsNRS2rbXrCkm9OGgEYG5IFz0iDMIRxnlCXm5VZVf2u9eTo/olOeU8BOvqSg7g19sz0FC1pJjdJA5mxpqZYCxyjS8gUpjaNqfMBmJfhoxbx+km5BNz/nxJlFgHuvVliErrRqIi5WpBAzhb1+iClncXXgd0FmjxsJhnQ26g5wLzU7faeNpL/OXMRtArFKjxHWtY+GjdEyjBp2KdpCRVvBoAvu3zb9JUL9Ze32Slgg1DfrklIqOzr1f5Aca4MBO9kFvnAntqdq3jgcPGgYBAVILMCDHCZJS+cMJ5tU89Kis3HAcjoc4LfyHdU/s=
2022-07-04T03:03:36.009 INFO:teuthology.kill:  smithi059.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDON88wCTxU8vJ57ov8Hmf2HA+YHT8LnzW5fPZjs+2s4Um0pThnK7BbCixMmzZEHB0hzCe0L9Mz1T8m2VaUKMDhNzeYk6g0UhlnqjxyHVktgXOjzRx9G8iezqUvNS1Bu9BehtJDOs4j/zFQ8EnWK55gHN441FUDutpBIhTpsztLKV/JIL/cO8tRNmwEiOv3xOqK5zycl6w26EFIYUVJLLHfMxHyFdHcJLIyaZtCw75Ex06hPIyjmg8Y842jrIuS/+wXYoV5pLwnJQYuGbJenFiFZsAzQeQqJkNj41MDpgaxwEiLzS0SvCERCvfZyUuYopMzT6UsWXYS34KXILQWF03E5mRieJ7UQhIXVB86aCQK1gdH/Snbr5UcBTHn5TQPY+UDaC5xI+snmZhpftOQLvwJ0/XsjafCSTmYHiBC6VHDEomT9jlE2GevVzz8/SCGuDQdSwFV6dHO+gAAGo3MHxBtCPwm6cqZWfSpMPbgApUV1li/kreXSYjNj0bzAiHQt4M=
2022-07-04T03:03:36.010 INFO:teuthology.kill:2022-07-04 03:03:36,005.005 INFO:teuthology.nuke:Checking targets against current locks
2022-07-04T03:03:36.068 INFO:teuthology.kill:2022-07-04 03:03:36,067.067 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-07-04T03:03:36.106 INFO:teuthology.kill:2022-07-04 03:03:36,105.105 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-07-04T03:03:36.181 INFO:teuthology.kill:2022-07-04 03:03:36,181.181 INFO:teuthology.orchestra.console:Power off smithi019
2022-07-04T03:03:36.227 INFO:teuthology.kill:2022-07-04 03:03:36,226.226 INFO:teuthology.orchestra.console:Power off smithi059
2022-07-04T03:03:44.589 INFO:teuthology.kill:2022-07-04 03:03:44,588.588 INFO:teuthology.orchestra.console:Power off for smithi019 completed
2022-07-04T03:03:44.606 INFO:teuthology.kill:2022-07-04 03:03:44,605.605 INFO:teuthology.orchestra.console:Power off for smithi059 completed
2022-07-04T03:03:44.748 INFO:teuthology.kill:2022-07-04 03:03:44,747.747 INFO:teuthology.lock.ops:unlocked: smithi019.front.sepia.ceph.com
2022-07-04T03:03:44.761 INFO:teuthology.kill:2022-07-04 03:03:44,761.761 INFO:teuthology.lock.ops:unlocked: smithi059.front.sepia.ceph.com
2022-07-04T03:05:45.047 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-07-04T03:05:45.187 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi019 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-07-03_03:31:04-rados-pacific-distro-default-smithi/6911110
2022-07-04T03:05:45.187 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi059 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-07-03_03:31:04-rados-pacific-distro-default-smithi/6911110