2022-07-03T00:13:45.770 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3 2022-07-03T00:13:45.922 INFO:teuthology.lock.ops:Start node 'smithi120.front.sepia.ceph.com' reimaging 2022-07-03T00:13:45.923 INFO:teuthology.lock.ops:Updating [smithi120.front.sepia.ceph.com]: reset os type and version on server 2022-07-03T00:13:45.924 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2022-07-03T00:13:45.948 INFO:teuthology.lock.ops:Node 'smithi120.front.sepia.ceph.com' reimaging is complete 2022-07-03T00:13:45.948 INFO:teuthology.lock.ops:Start node 'smithi196.front.sepia.ceph.com' reimaging 2022-07-03T00:13:45.949 INFO:teuthology.lock.ops:Updating [smithi196.front.sepia.ceph.com]: reset os type and version on server 2022-07-03T00:13:45.949 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2022-07-03T00:13:45.973 INFO:teuthology.lock.ops:Node 'smithi196.front.sepia.ceph.com' reimaging is complete 2022-07-03T00:13:46.165 INFO:teuthology.provision.fog.smithi120:Scheduling deploy of ubuntu 20.04 2022-07-03T00:13:46.179 INFO:teuthology.provision.fog.smithi196:Scheduling deploy of ubuntu 20.04 2022-07-03T00:13:46.468 INFO:teuthology.orchestra.console:Power off smithi120 2022-07-03T00:13:46.483 INFO:teuthology.orchestra.console:Power off smithi196 2022-07-03T00:13:46.615 INFO:teuthology.orchestra.console:Power off for smithi120 completed 2022-07-03T00:13:46.629 INFO:teuthology.orchestra.console:Power off for smithi196 completed 2022-07-03T00:13:46.717 INFO:teuthology.orchestra.console:Power on smithi120 2022-07-03T00:13:46.730 INFO:teuthology.orchestra.console:Power on smithi196 2022-07-03T00:13:50.971 INFO:teuthology.orchestra.console:Power on for smithi120 completed 2022-07-03T00:13:50.978 INFO:teuthology.orchestra.console:Power on for smithi196 completed 2022-07-03T00:13:51.073 INFO:teuthology.provision.fog.smithi120:Waiting for deploy to finish 2022-07-03T00:13:51.080 INFO:teuthology.provision.fog.smithi196:Waiting for deploy to finish 2022-07-03T00:18:48.250 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-07-03T00:18:48.774 INFO:teuthology.provision.fog.smithi120:Node is ready 2022-07-03T00:18:48.781 INFO:teuthology.orchestra.run.smithi120.stdout:smithi120.front.sepia.ceph.com 2022-07-03T00:18:48.788 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.15.120 smithi120.front.sepia.ceph.com smithi120 2022-07-03T00:18:48.935 INFO:teuthology.provision.fog.smithi120:Deploy complete! 2022-07-03T00:18:54.238 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-07-03T00:18:54.778 INFO:teuthology.provision.fog.smithi196:Node is ready 2022-07-03T00:18:54.827 INFO:teuthology.orchestra.run.smithi196.stdout:smithi196.front.sepia.ceph.com 2022-07-03T00:18:54.834 INFO:teuthology.orchestra.run.smithi196.stdout:172.21.15.196 smithi196.front.sepia.ceph.com smithi196 2022-07-03T00:18:55.122 INFO:teuthology.provision.fog.smithi196:Deploy complete! 2022-07-03T00:18:55.362 INFO:teuthology.lock.ops:Checking smithi120.front.sepia.ceph.com 2022-07-03T00:18:55.364 INFO:teuthology.lock.ops:New key found. Updating... 2022-07-03T00:18:55.393 INFO:teuthology.lock.ops:Checking smithi196.front.sepia.ceph.com 2022-07-03T00:18:55.393 INFO:teuthology.lock.ops:New key found. Updating... 2022-07-03T00:18:55.418 INFO:teuthology.lock.ops:Updating [smithi120.front.sepia.ceph.com]: set os type and version on server 2022-07-03T00:18:55.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2022-07-03T00:18:55.663 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64 2022-07-03T00:18:55.670 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="Ubuntu" 2022-07-03T00:18:55.670 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-03T00:18:55.671 INFO:teuthology.orchestra.run.smithi120.stdout:ID=ubuntu 2022-07-03T00:18:55.671 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE=debian 2022-07-03T00:18:55.671 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-03T00:18:55.671 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="20.04" 2022-07-03T00:18:55.672 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-03T00:18:55.672 INFO:teuthology.orchestra.run.smithi120.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-03T00:18:55.672 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-03T00:18:55.672 INFO:teuthology.orchestra.run.smithi120.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-03T00:18:55.672 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_CODENAME=focal 2022-07-03T00:18:55.673 INFO:teuthology.orchestra.run.smithi120.stdout:UBUNTU_CODENAME=focal 2022-07-03T00:18:55.674 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2022-07-03T00:18:55.699 INFO:teuthology.lock.ops:Updating [smithi196.front.sepia.ceph.com]: set os type and version on server 2022-07-03T00:18:55.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2022-07-03T00:18:55.930 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2022-07-03T00:18:55.980 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="Ubuntu" 2022-07-03T00:18:55.981 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-03T00:18:55.981 INFO:teuthology.orchestra.run.smithi196.stdout:ID=ubuntu 2022-07-03T00:18:55.982 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE=debian 2022-07-03T00:18:55.982 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-03T00:18:55.983 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="20.04" 2022-07-03T00:18:55.984 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-03T00:18:55.984 INFO:teuthology.orchestra.run.smithi196.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-03T00:18:55.984 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-03T00:18:55.985 INFO:teuthology.orchestra.run.smithi196.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-03T00:18:55.985 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_CODENAME=focal 2022-07-03T00:18:55.986 INFO:teuthology.orchestra.run.smithi196.stdout:UBUNTU_CODENAME=focal 2022-07-03T00:18:55.987 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2022-07-03T00:18:56.171 INFO:teuthology.dispatcher.supervisor:Running job 6910673 2022-07-03T00:18:56.171 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/bin/teuthology -v --owner scheduled_rishabh@teuthology --archive /home/teuthworker/archive/rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi/6910673 --name rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi --description fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} -- /home/teuthworker/archive/rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi/6910673/orig.config.yaml 2022-07-03T00:18:56.176 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rishabh-2022-07-02_14:14:52-fs-wip-rishabh-testing-20220702-1631-testing-default-smithi/6910673 2022-07-03T00:18:56.177 INFO:teuthology.dispatcher.supervisor:Job PID: 9028 2022-07-03T00:18:56.178 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-07-03T12:19:53.831 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-07-03T12:19:54.051 INFO:teuthology.kill:Killing Pids: {9028} 2022-07-03T12:19:55.479 INFO:teuthology.task.internal:roles: ubuntu@smithi120.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2022-07-03T12:19:55.480 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2022-07-03T12:19:55.480 INFO:teuthology.misc:Compressing logs... 2022-07-03T12:19:55.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2022-07-03T12:20:10.547 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner 2022-07-03T12:20:10.549 ERROR:paramiko.transport:Traceback (most recent call last): 2022-07-03T12:20:10.549 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2271, in _check_banner 2022-07-03T12:20:10.550 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2022-07-03T12:20:10.550 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 380, in readline 2022-07-03T12:20:10.550 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2022-07-03T12:20:10.551 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 622, in _read_timeout 2022-07-03T12:20:10.551 ERROR:paramiko.transport: raise socket.timeout() 2022-07-03T12:20:10.552 ERROR:paramiko.transport:socket.timeout 2022-07-03T12:20:10.552 ERROR:paramiko.transport: 2022-07-03T12:20:10.552 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2022-07-03T12:20:10.553 ERROR:paramiko.transport: 2022-07-03T12:20:10.553 ERROR:paramiko.transport:Traceback (most recent call last): 2022-07-03T12:20:10.553 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2094, in run 2022-07-03T12:20:10.553 ERROR:paramiko.transport: self._check_banner() 2022-07-03T12:20:10.554 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2276, in _check_banner 2022-07-03T12:20:10.554 ERROR:paramiko.transport: "Error reading SSH protocol banner" + str(e) 2022-07-03T12:20:10.554 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2022-07-03T12:20:10.555 ERROR:paramiko.transport: 2022-07-03T12:20:10.556 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/dispatcher/supervisor.py", line 290, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/remote.py", line 509, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi120 2022-07-03T12:20:10.623 INFO:teuthology.kill:No teuthology processes running 2022-07-03T12:20:12.052 INFO:teuthology.kill:Nuking machines: ['smithi120', 'smithi196'] 2022-07-03T12:20:13.205 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-07-03T12:20:13.206 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-07-03T12:20:13.206 INFO:teuthology.kill:2022-07-03 12:20:13,204.204 INFO:teuthology.nuke:targets: 2022-07-03T12:20:13.207 INFO:teuthology.kill: smithi120.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQ+Tib02Rs21Kig+NTOlCXhikRkrzSS9jGQ0kTIIG52T4pQlJHG1L+8rexta1cSJ5ccWj352qnAt7WO6KvA3rT2AFzcESJQljWU8c7YqE64PdDTanJ76BxaJJpOn6g8rfIDaywuECGfrN6imNDVkMdK/e4kWVhovGAwe9+/YHYMaJ5T9fUMzeKzk+8i7lAhZ1QtTyH2naTC8k84miH4EUDuGuCuyOvjOO/j4CXonwFgp2R97uC4/wlrLEA4S2YY6vSbO+h4TWDW5xHi8YZwcsqOqcTiATagHqYFO5TNy19W8qSuOOdkcBX1CezBwF3E4ppike5huy93jcO4TzuC7RzeMt/yuQO75QbDaE+pDYh/Piu8PlYrNxdr6anTQMNPp38i2Fwi9+gJIYKJT2AzqmQhgmzgJkgK1xAkSVbir+GuFE/q2+VTIiSRsnwx5sVCyVaF1hhmKHgahy1Y0xUWsOqFE+0j0mwBODzLUmcTcVSxwgH+YUh4qpUp/iP2889LtU= 2022-07-03T12:20:13.207 INFO:teuthology.kill: smithi196.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAHDupo2oB32cRSUoLldCBntFoW/HGkBKBhWJeHRZF8FrLVilcbwOdfd8s18/ti3QM5kFbo2qaZLdyGY49aLvjsxvRrpZOvh0FxSKl5cfT1TSG09ds9H0gHWhoHibgxOc45EhtppJCJ6ojcucWhwQoG7u1swyxennQ2Zf2BIT4NTu23SqZiaTSznSds+ynoYg6j/efgQ+mZ1xzdZ7VT8rQqWthDdmsBElMSHEeKgYjsasNifhTUzhryMf4m45fbAz5u5YtxCepOFtw6qHadkD22qj1fs8xkqfsQPNUR1soFdLjuUtVPsInnp7/z9FZ7qVNoTw93YZV0Lb9QKuj7DieNlbDBqlTL5tcPFGB69CP7tROlh8eHufMO2M1+FozwxjRI4SgFbPPUYLNw+zlfODm+5oE3oF4pvZ0Jg93pSd7yJFxBEIwcBC7Nxb1a8HgAUWcK1ImH7940G1C45rKjerqO/GcaAtj0cMkcZyhSS06CbHBh/qWDU4Ml+DvbqK3Cz0= 2022-07-03T12:20:13.208 INFO:teuthology.kill:2022-07-03 12:20:13,205.205 INFO:teuthology.nuke:Checking targets against current locks 2022-07-03T12:20:13.269 INFO:teuthology.kill:2022-07-03 12:20:13,268.268 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-03T12:20:13.296 INFO:teuthology.kill:2022-07-03 12:20:13,295.295 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-03T12:20:13.367 INFO:teuthology.kill:2022-07-03 12:20:13,367.367 INFO:teuthology.orchestra.console:Power off smithi120 2022-07-03T12:20:13.408 INFO:teuthology.kill:2022-07-03 12:20:13,408.408 INFO:teuthology.orchestra.console:Power off smithi196 2022-07-03T12:20:25.877 INFO:teuthology.kill:2022-07-03 12:20:25,876.876 INFO:teuthology.orchestra.console:Power off for smithi120 completed 2022-07-03T12:20:25.888 INFO:teuthology.kill:2022-07-03 12:20:25,888.888 INFO:teuthology.orchestra.console:Power off for smithi196 completed 2022-07-03T12:20:26.043 INFO:teuthology.kill:2022-07-03 12:20:26,041.041 INFO:teuthology.lock.ops:unlocked: smithi120.front.sepia.ceph.com 2022-07-03T12:20:26.046 INFO:teuthology.kill:2022-07-03 12:20:26,046.046 INFO:teuthology.lock.ops:unlocked: smithi196.front.sepia.ceph.com 2022-07-03T12:22:26.454 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-07-03T12:22:26.597 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi120 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-07-02_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6910895 2022-07-03T12:22:26.597 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi196 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-07-02_14:22:03-upgrade:octopus-x-pacific-distro-default-smithi/6910895