2023-01-02T18:01:57.570 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2023-01-02T18:01:57.723 INFO:teuthology.lock.ops:Start node 'smithi124.front.sepia.ceph.com' reimaging 2023-01-02T18:01:57.724 INFO:teuthology.lock.ops:Updating [smithi124.front.sepia.ceph.com]: reset os type and version on server 2023-01-02T18:01:57.724 INFO:teuthology.lock.ops:Updating smithi124.front.sepia.ceph.com on lock server 2023-01-02T18:01:57.738 INFO:teuthology.lock.ops:Node 'smithi124.front.sepia.ceph.com' reimaging is complete 2023-01-02T18:01:57.738 INFO:teuthology.lock.ops:Start node 'smithi137.front.sepia.ceph.com' reimaging 2023-01-02T18:01:57.739 INFO:teuthology.lock.ops:Updating [smithi137.front.sepia.ceph.com]: reset os type and version on server 2023-01-02T18:01:57.739 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2023-01-02T18:01:57.753 INFO:teuthology.lock.ops:Node 'smithi137.front.sepia.ceph.com' reimaging is complete 2023-01-02T18:01:57.754 INFO:teuthology.lock.ops:Start node 'smithi153.front.sepia.ceph.com' reimaging 2023-01-02T18:01:57.754 INFO:teuthology.lock.ops:Updating [smithi153.front.sepia.ceph.com]: reset os type and version on server 2023-01-02T18:01:57.754 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2023-01-02T18:01:57.768 INFO:teuthology.lock.ops:Node 'smithi153.front.sepia.ceph.com' reimaging is complete 2023-01-02T18:01:57.768 INFO:teuthology.lock.ops:Start node 'smithi175.front.sepia.ceph.com' reimaging 2023-01-02T18:01:57.768 INFO:teuthology.lock.ops:Updating [smithi175.front.sepia.ceph.com]: reset os type and version on server 2023-01-02T18:01:57.769 INFO:teuthology.lock.ops:Updating smithi175.front.sepia.ceph.com on lock server 2023-01-02T18:01:57.782 INFO:teuthology.lock.ops:Node 'smithi175.front.sepia.ceph.com' reimaging is complete 2023-01-02T18:01:57.894 INFO:teuthology.provision.fog.smithi137:Scheduling deploy of ubuntu 20.04 2023-01-02T18:01:57.915 INFO:teuthology.provision.fog.smithi124:Scheduling deploy of ubuntu 20.04 2023-01-02T18:01:57.918 INFO:teuthology.provision.fog.smithi153:Scheduling deploy of ubuntu 20.04 2023-01-02T18:01:57.924 INFO:teuthology.provision.fog.smithi175:Scheduling deploy of ubuntu 20.04 2023-01-02T18:01:58.137 INFO:teuthology.orchestra.console:Power off smithi137 2023-01-02T18:01:58.177 INFO:teuthology.orchestra.console:Power off smithi175 2023-01-02T18:01:58.189 INFO:teuthology.orchestra.console:Power off smithi124 2023-01-02T18:01:58.198 INFO:teuthology.orchestra.console:Power off smithi153 2023-01-02T18:01:58.305 INFO:teuthology.orchestra.console:Power off for smithi137 completed 2023-01-02T18:01:58.325 INFO:teuthology.orchestra.console:Power off for smithi175 completed 2023-01-02T18:01:58.330 INFO:teuthology.orchestra.console:Power off for smithi124 completed 2023-01-02T18:01:58.338 INFO:teuthology.orchestra.console:Power off for smithi153 completed 2023-01-02T18:01:58.407 INFO:teuthology.orchestra.console:Power on smithi137 2023-01-02T18:01:58.432 INFO:teuthology.orchestra.console:Power on smithi175 2023-01-02T18:01:58.439 INFO:teuthology.orchestra.console:Power on smithi124 2023-01-02T18:01:58.445 INFO:teuthology.orchestra.console:Power on smithi153 2023-01-02T18:02:02.664 INFO:teuthology.orchestra.console:Power on for smithi137 completed 2023-01-02T18:02:02.765 INFO:teuthology.provision.fog.smithi137:Waiting for deploy to finish 2023-01-02T18:02:06.793 INFO:teuthology.orchestra.console:Power on for smithi175 completed 2023-01-02T18:02:06.803 INFO:teuthology.orchestra.console:Power on for smithi153 completed 2023-01-02T18:02:06.895 INFO:teuthology.provision.fog.smithi175:Waiting for deploy to finish 2023-01-02T18:02:06.897 INFO:teuthology.orchestra.console:Power on for smithi124 completed 2023-01-02T18:02:06.905 INFO:teuthology.provision.fog.smithi153:Waiting for deploy to finish 2023-01-02T18:02:06.998 INFO:teuthology.provision.fog.smithi124:Waiting for deploy to finish 2023-01-02T18:05:56.213 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.175 2023-01-02T18:06:05.301 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.175 2023-01-02T18:06:07.225 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.137 2023-01-02T18:06:11.317 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.153 2023-01-02T18:06:11.318 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.124 2023-01-02T18:06:14.357 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.175 2023-01-02T18:06:16.277 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.137 2023-01-02T18:06:20.377 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.153 2023-01-02T18:06:20.378 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.124 2023-01-02T18:06:23.417 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.175 2023-01-02T18:06:25.337 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.137 2023-01-02T18:06:29.433 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.153 2023-01-02T18:06:29.461 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.124 2023-01-02T18:06:38.517 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.153 2023-01-02T18:06:38.518 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.124 2023-01-02T18:06:47.573 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.153 2023-01-02T18:06:47.574 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.124 2023-01-02T18:07:25.459 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-02T18:07:26.304 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-02T18:07:29.422 WARNING:teuthology.provision.fog:timed out 2023-01-02T18:07:31.341 WARNING:teuthology.provision.fog:timed out 2023-01-02T18:07:35.630 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-02T18:07:36.149 INFO:teuthology.provision.fog.smithi175:Node is ready 2023-01-02T18:07:36.155 INFO:teuthology.orchestra.run.smithi175.stdout:smithi175.front.sepia.ceph.com 2023-01-02T18:07:36.162 INFO:teuthology.orchestra.run.smithi175.stdout:172.21.15.175 smithi175.front.sepia.ceph.com smithi175 2023-01-02T18:07:36.389 INFO:teuthology.provision.fog.smithi175:Deploy complete! 2023-01-02T18:07:37.547 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-02T18:07:38.061 INFO:teuthology.provision.fog.smithi137:Node is ready 2023-01-02T18:07:38.067 INFO:teuthology.orchestra.run.smithi137.stdout:smithi137.front.sepia.ceph.com 2023-01-02T18:07:38.077 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.15.137 smithi137.front.sepia.ceph.com smithi137 2023-01-02T18:07:38.213 INFO:teuthology.provision.fog.smithi137:Deploy complete! 2023-01-02T18:07:41.271 INFO:teuthology.provision.fog.smithi124:Node is ready 2023-01-02T18:07:41.278 INFO:teuthology.orchestra.run.smithi124.stdout:smithi129 2023-01-02T18:07:41.286 INFO:teuthology.orchestra.run.smithi124.stdout:172.21.15.129 smithi129 smithi129 2023-01-02T18:07:41.454 INFO:teuthology.provision.fog.smithi124:Deploy complete! 2023-01-02T18:07:48.132 INFO:teuthology.provision.fog.smithi153:Node is ready 2023-01-02T18:07:48.139 INFO:teuthology.orchestra.run.smithi153.stdout:smithi153.front.sepia.ceph.com 2023-01-02T18:07:48.147 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153 smithi153.front.sepia.ceph.com smithi153 2023-01-02T18:07:48.322 INFO:teuthology.provision.fog.smithi153:Deploy complete! 2023-01-02T18:07:48.697 INFO:teuthology.lock.ops:Checking smithi124.front.sepia.ceph.com 2023-01-02T18:07:48.698 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-02T18:07:48.718 INFO:teuthology.lock.ops:Checking smithi137.front.sepia.ceph.com 2023-01-02T18:07:48.718 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-02T18:07:48.732 INFO:teuthology.lock.ops:Checking smithi153.front.sepia.ceph.com 2023-01-02T18:07:48.733 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-02T18:07:48.750 INFO:teuthology.lock.ops:Checking smithi175.front.sepia.ceph.com 2023-01-02T18:07:48.750 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-02T18:07:48.764 INFO:teuthology.lock.ops:Updating [smithi124.front.sepia.ceph.com]: set os type and version on server 2023-01-02T18:07:48.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-02T18:07:48.941 INFO:teuthology.orchestra.run.smithi124.stdout:x86_64 2023-01-02T18:07:48.948 INFO:teuthology.orchestra.run.smithi124.stdout:NAME="Ubuntu" 2023-01-02T18:07:48.948 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-01-02T18:07:48.949 INFO:teuthology.orchestra.run.smithi124.stdout:ID=ubuntu 2023-01-02T18:07:48.949 INFO:teuthology.orchestra.run.smithi124.stdout:ID_LIKE=debian 2023-01-02T18:07:48.949 INFO:teuthology.orchestra.run.smithi124.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-01-02T18:07:48.949 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION_ID="20.04" 2023-01-02T18:07:48.950 INFO:teuthology.orchestra.run.smithi124.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-02T18:07:48.950 INFO:teuthology.orchestra.run.smithi124.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-02T18:07:48.950 INFO:teuthology.orchestra.run.smithi124.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-02T18:07:48.950 INFO:teuthology.orchestra.run.smithi124.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-02T18:07:48.950 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION_CODENAME=focal 2023-01-02T18:07:48.951 INFO:teuthology.orchestra.run.smithi124.stdout:UBUNTU_CODENAME=focal 2023-01-02T18:07:48.951 INFO:teuthology.lock.ops:Updating smithi124.front.sepia.ceph.com on lock server 2023-01-02T18:07:48.967 INFO:teuthology.lock.ops:Updating [smithi137.front.sepia.ceph.com]: set os type and version on server 2023-01-02T18:07:48.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2023-01-02T18:07:49.152 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2023-01-02T18:07:49.200 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="Ubuntu" 2023-01-02T18:07:49.200 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-01-02T18:07:49.201 INFO:teuthology.orchestra.run.smithi137.stdout:ID=ubuntu 2023-01-02T18:07:49.201 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE=debian 2023-01-02T18:07:49.201 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-01-02T18:07:49.201 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="20.04" 2023-01-02T18:07:49.201 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-02T18:07:49.202 INFO:teuthology.orchestra.run.smithi137.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-02T18:07:49.202 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-02T18:07:49.202 INFO:teuthology.orchestra.run.smithi137.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-02T18:07:49.202 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_CODENAME=focal 2023-01-02T18:07:49.202 INFO:teuthology.orchestra.run.smithi137.stdout:UBUNTU_CODENAME=focal 2023-01-02T18:07:49.203 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2023-01-02T18:07:49.216 INFO:teuthology.lock.ops:Updating [smithi153.front.sepia.ceph.com]: set os type and version on server 2023-01-02T18:07:49.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-01-02T18:07:49.437 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2023-01-02T18:07:49.444 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="Ubuntu" 2023-01-02T18:07:49.445 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-01-02T18:07:49.445 INFO:teuthology.orchestra.run.smithi153.stdout:ID=ubuntu 2023-01-02T18:07:49.445 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE=debian 2023-01-02T18:07:49.446 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-01-02T18:07:49.446 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="20.04" 2023-01-02T18:07:49.446 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-02T18:07:49.446 INFO:teuthology.orchestra.run.smithi153.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-02T18:07:49.446 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-02T18:07:49.447 INFO:teuthology.orchestra.run.smithi153.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-02T18:07:49.447 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_CODENAME=focal 2023-01-02T18:07:49.447 INFO:teuthology.orchestra.run.smithi153.stdout:UBUNTU_CODENAME=focal 2023-01-02T18:07:49.448 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2023-01-02T18:07:49.462 INFO:teuthology.lock.ops:Updating [smithi175.front.sepia.ceph.com]: set os type and version on server 2023-01-02T18:07:49.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2023-01-02T18:07:49.720 INFO:teuthology.orchestra.run.smithi175.stdout:x86_64 2023-01-02T18:07:49.726 INFO:teuthology.orchestra.run.smithi175.stdout:NAME="Ubuntu" 2023-01-02T18:07:49.727 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-01-02T18:07:49.727 INFO:teuthology.orchestra.run.smithi175.stdout:ID=ubuntu 2023-01-02T18:07:49.727 INFO:teuthology.orchestra.run.smithi175.stdout:ID_LIKE=debian 2023-01-02T18:07:49.727 INFO:teuthology.orchestra.run.smithi175.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-01-02T18:07:49.728 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION_ID="20.04" 2023-01-02T18:07:49.728 INFO:teuthology.orchestra.run.smithi175.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-02T18:07:49.728 INFO:teuthology.orchestra.run.smithi175.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-02T18:07:49.728 INFO:teuthology.orchestra.run.smithi175.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-02T18:07:49.728 INFO:teuthology.orchestra.run.smithi175.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-02T18:07:49.729 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION_CODENAME=focal 2023-01-02T18:07:49.729 INFO:teuthology.orchestra.run.smithi175.stdout:UBUNTU_CODENAME=focal 2023-01-02T18:07:49.729 INFO:teuthology.lock.ops:Updating smithi175.front.sepia.ceph.com on lock server 2023-01-02T18:07:49.805 INFO:teuthology.dispatcher.supervisor:Running job 7131282 2023-01-02T18:07:49.805 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-01-02_16:43:22-powercycle-pacific_16.2.11_RC4-distro-default-smithi/7131282 --name yuriw-2023-01-02_16:43:22-powercycle-pacific_16.2.11_RC4-distro-default-smithi --description powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-comp-zlib powercycle/default supported-all-distro/ubuntu_latest tasks/cfuse_workunit_misc thrashosds-health} -- /home/teuthworker/archive/yuriw-2023-01-02_16:43:22-powercycle-pacific_16.2.11_RC4-distro-default-smithi/7131282/orig.config.yaml 2023-01-02T18:07:49.808 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-01-02_16:43:22-powercycle-pacific_16.2.11_RC4-distro-default-smithi/7131282 2023-01-02T18:07:49.809 INFO:teuthology.dispatcher.supervisor:Job PID: 13982 2023-01-02T18:07:49.809 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-01-03T06:08:32.964 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:08:33.119 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:08:33.757 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:08:33.758 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:08:33.758 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:08:33.758 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:08:33.758 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:08:33.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:08:33.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2023-01-03T06:08:34.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2023-01-03T06:08:34.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2023-01-03T06:08:34.589 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:08:34.794 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:08:53.638 INFO:teuthology.orchestra.run.smithi124.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping 2023-01-03T06:08:58.489 INFO:teuthology.orchestra.run.smithi153.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping 2023-01-03T06:09:00.070 INFO:teuthology.orchestra.run.smithi124.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2023-01-03T06:09:03.072 INFO:teuthology.orchestra.run.smithi124.stderr:gzip: /var/log/ceph/ceph-mgr.y.log: file size changed while zipping 2023-01-03T06:09:07.915 INFO:teuthology.orchestra.run.smithi175.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping 2023-01-03T06:09:11.339 INFO:teuthology.orchestra.run.smithi124.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2023-01-03T06:09:27.429 INFO:teuthology.orchestra.run.smithi137.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping 2023-01-03T06:10:01.664 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:10:02.304 INFO:teuthology.kill:Nuking machines: ['smithi124', 'smithi137', 'smithi153', 'smithi175'] 2023-01-03T06:10:02.810 INFO:teuthology.kill:2023-01-03 06:10:02,810.810 INFO:teuthology.nuke:targets: 2023-01-03T06:10:02.811 INFO:teuthology.kill: smithi124.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= 2023-01-03T06:10:02.811 INFO:teuthology.kill: smithi137.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= 2023-01-03T06:10:02.812 INFO:teuthology.kill: smithi153.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= 2023-01-03T06:10:02.812 INFO:teuthology.kill: smithi175.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= 2023-01-03T06:10:02.813 INFO:teuthology.kill:2023-01-03 06:10:02,810.810 INFO:teuthology.nuke:Checking targets against current locks 2023-01-03T06:10:02.850 INFO:teuthology.kill:2023-01-03 06:10:02,850.850 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-03T06:10:02.872 INFO:teuthology.kill:2023-01-03 06:10:02,872.872 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-03T06:10:02.894 INFO:teuthology.kill:2023-01-03 06:10:02,894.894 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-03T06:10:02.921 INFO:teuthology.kill:2023-01-03 06:10:02,920.920 INFO:teuthology.orchestra.console:Power off smithi124 2023-01-03T06:10:02.930 INFO:teuthology.kill:2023-01-03 06:10:02,929.929 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-03T06:10:02.951 INFO:teuthology.kill:2023-01-03 06:10:02,951.951 INFO:teuthology.orchestra.console:Power off smithi137 2023-01-03T06:10:02.984 INFO:teuthology.kill:2023-01-03 06:10:02,984.984 INFO:teuthology.orchestra.console:Power off smithi153 2023-01-03T06:10:03.014 INFO:teuthology.kill:2023-01-03 06:10:03,014.014 INFO:teuthology.orchestra.console:Power off smithi175 2023-01-03T06:10:11.290 INFO:teuthology.kill:2023-01-03 06:10:11,289.289 INFO:teuthology.orchestra.console:Power off for smithi124 completed 2023-01-03T06:10:11.308 INFO:teuthology.kill:2023-01-03 06:10:11,308.308 INFO:teuthology.orchestra.console:Power off for smithi137 completed 2023-01-03T06:10:11.339 INFO:teuthology.kill:2023-01-03 06:10:11,338.338 INFO:teuthology.orchestra.console:Power off for smithi153 completed 2023-01-03T06:10:11.442 INFO:teuthology.kill:2023-01-03 06:10:11,442.442 INFO:teuthology.lock.ops:unlocked: smithi124.front.sepia.ceph.com 2023-01-03T06:10:11.449 INFO:teuthology.kill:2023-01-03 06:10:11,449.449 INFO:teuthology.lock.ops:unlocked: smithi137.front.sepia.ceph.com 2023-01-03T06:10:11.475 INFO:teuthology.kill:2023-01-03 06:10:11,475.475 INFO:teuthology.lock.ops:unlocked: smithi153.front.sepia.ceph.com 2023-01-03T06:10:15.479 INFO:teuthology.kill:2023-01-03 06:10:15,479.479 INFO:teuthology.orchestra.console:Power off for smithi175 completed 2023-01-03T06:10:15.619 INFO:teuthology.kill:2023-01-03 06:10:15,619.619 INFO:teuthology.lock.ops:unlocked: smithi175.front.sepia.ceph.com 2023-01-03T06:12:15.826 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:12:15.863 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:12:16.497 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:12:16.497 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:12:16.498 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:12:16.498 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:12:16.498 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:12:16.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:12:35.062 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:12:35.102 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:12:35.767 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:14:35.918 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:14:35.959 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:14:36.591 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:14:36.591 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:14:36.591 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:14:36.592 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:14:36.592 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:14:36.592 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:14:39.669 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:14:39.707 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:14:40.318 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:16:40.460 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:16:40.500 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:16:41.149 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:16:41.150 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:16:41.150 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:16:41.150 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:16:41.150 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:16:41.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:16:44.213 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:16:44.252 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:16:44.863 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:18:44.984 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:18:45.020 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:18:45.659 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:18:45.659 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:18:45.659 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:18:45.660 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:18:45.660 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:18:45.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:18:48.726 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:18:48.769 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:18:49.383 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:20:49.530 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:20:49.568 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:20:50.260 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:20:50.260 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:20:50.260 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:20:50.261 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:20:50.261 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:20:50.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:20:53.334 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:20:53.376 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:20:53.987 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:22:54.135 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:22:54.174 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:22:54.831 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:22:54.831 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:22:54.831 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:22:54.832 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:22:54.832 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:22:54.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:22:57.909 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:22:57.950 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:22:58.566 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:24:58.696 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:24:58.769 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:24:59.397 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:24:59.398 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:24:59.398 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:24:59.398 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:24:59.398 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:24:59.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:25:02.453 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:25:02.496 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:25:03.183 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:27:03.286 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:27:03.330 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:27:04.046 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:27:04.046 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:27:04.047 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:27:04.047 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:27:04.047 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:27:04.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:27:07.126 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:27:07.168 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:27:07.937 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:29:08.080 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:29:08.136 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:29:08.847 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:29:08.848 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:29:08.848 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:29:08.848 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:29:08.848 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:29:08.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:29:11.926 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:29:11.968 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:29:12.684 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:31:12.831 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:31:12.878 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:31:13.593 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:31:13.594 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:31:13.594 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:31:13.594 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:31:13.595 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:31:13.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:31:16.661 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:31:16.706 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:31:17.384 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:33:17.527 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:33:17.572 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:33:18.288 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:33:18.288 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:33:18.289 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:33:18.289 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:33:18.289 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:33:18.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:33:21.365 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:33:21.406 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:33:22.092 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:35:22.231 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:35:22.281 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:35:23.016 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:35:23.017 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:35:23.018 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:35:23.018 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:35:23.018 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:35:23.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:35:26.105 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:35:26.152 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:35:26.852 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:37:26.997 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:37:27.041 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:37:27.776 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:37:27.777 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:37:27.777 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:37:27.778 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:37:27.778 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:37:27.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:37:30.841 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:37:30.882 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:37:31.570 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:39:31.713 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:39:31.757 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:39:32.498 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:39:32.499 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:39:32.499 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:39:32.500 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:39:32.500 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:39:32.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:39:35.573 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:39:35.621 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:39:36.318 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:41:36.421 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:41:36.467 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:41:37.185 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:41:37.186 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:41:37.186 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:41:37.186 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:41:37.186 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:41:37.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:41:40.245 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:41:40.287 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:41:40.957 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:43:41.108 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:43:41.151 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:43:41.853 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:43:41.854 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:43:41.854 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:43:41.854 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:43:41.855 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:43:41.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:43:44.917 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:43:44.959 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:43:45.645 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:45:45.749 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:45:45.792 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:45:46.504 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:45:46.504 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:45:46.504 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:45:46.505 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:45:46.505 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:45:46.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:45:49.557 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:45:49.598 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:45:50.298 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:47:50.445 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:47:50.491 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:47:51.194 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:47:51.194 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:47:51.195 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:47:51.195 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:47:51.195 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:47:51.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:47:54.261 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:47:54.305 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:47:54.989 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:49:55.082 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:49:55.125 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:49:55.820 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:49:55.821 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:49:55.821 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:49:55.821 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:49:55.822 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:49:55.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:49:58.901 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:49:58.944 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:49:59.605 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:51:59.757 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:51:59.800 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:52:00.493 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:52:00.494 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:52:00.494 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:52:00.494 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:52:00.495 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:52:00.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:52:03.573 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:52:03.615 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:52:04.291 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:54:04.421 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:54:04.463 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:54:05.158 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:54:05.159 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:54:05.159 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:54:05.159 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:54:05.160 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:54:05.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:54:08.213 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:54:08.256 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:54:08.916 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:56:09.014 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:56:09.054 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:56:09.729 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:56:09.730 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:56:09.730 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:56:09.730 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:56:09.730 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:56:09.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:56:12.789 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:56:12.826 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:56:13.427 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:58:13.556 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:58:13.593 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:58:14.209 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T06:58:14.210 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T06:58:14.210 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T06:58:14.210 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T06:58:14.210 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:58:14.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T06:58:17.270 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T06:58:17.309 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T06:58:17.920 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:00:18.008 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:00:18.047 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:00:18.658 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:00:18.659 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:00:18.659 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:00:18.659 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:00:18.659 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:00:18.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:00:21.717 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:00:21.758 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:00:22.383 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:02:22.529 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:02:22.569 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:02:23.219 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:02:23.220 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:02:23.220 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:02:23.220 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:02:23.221 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:02:23.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:02:26.293 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:02:26.332 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:02:26.940 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:04:27.031 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:04:27.068 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:04:27.707 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:04:27.708 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:04:27.708 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:04:27.708 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:04:27.709 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:04:27.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:04:30.773 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:04:30.813 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:04:31.449 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:06:31.597 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:06:31.640 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:06:32.263 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:06:32.264 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:06:32.264 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:06:32.265 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:06:32.265 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:06:32.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:06:35.317 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:06:35.357 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:06:35.956 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:08:36.074 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:08:36.113 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:08:36.764 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:08:36.765 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:08:36.765 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:08:36.765 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:08:36.765 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:08:36.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:08:39.829 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:08:39.865 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:08:40.468 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:10:40.610 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:10:40.651 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:10:41.307 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:10:41.307 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:10:41.307 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:10:41.308 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:10:41.308 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:10:41.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:10:44.374 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:10:44.413 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:10:45.028 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:12:45.085 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:12:45.180 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:12:45.800 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:12:45.800 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:12:45.800 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:12:45.801 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:12:45.801 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:12:45.801 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:12:48.854 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:12:48.892 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:12:49.491 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:14:49.626 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:14:49.677 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:14:50.299 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:14:50.300 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:14:50.300 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:14:50.300 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:14:50.301 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:14:50.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:14:53.366 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:14:53.405 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:14:54.047 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:16:54.172 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:16:54.213 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:16:54.837 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:16:54.837 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:16:54.838 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:16:54.838 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:16:54.838 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:16:54.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:16:57.910 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:16:57.950 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:16:58.559 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:18:58.617 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:18:58.657 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:18:59.286 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:18:59.286 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:18:59.287 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:18:59.287 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:18:59.287 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:18:59.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:19:02.358 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:19:02.397 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:19:03.005 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:21:03.126 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:21:03.182 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:21:03.813 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:21:03.813 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:21:03.814 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:21:03.814 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:21:03.814 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:21:03.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:21:06.869 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:21:06.909 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:21:07.531 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:23:07.638 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:23:07.678 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:23:08.299 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:23:08.300 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:23:08.300 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:23:08.300 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:23:08.300 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:23:08.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:23:11.382 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:23:11.421 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:23:12.035 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:25:12.137 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:25:12.174 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:25:12.821 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:25:12.822 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:25:12.822 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:25:12.822 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:25:12.822 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:25:12.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:25:15.893 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:25:15.934 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:25:16.548 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:27:16.664 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:27:16.704 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:27:17.359 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:27:17.359 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:27:17.360 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:27:17.360 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:27:17.360 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:27:17.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:27:20.437 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:27:20.477 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:27:21.092 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:29:21.244 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:29:21.367 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:29:22.030 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:29:22.031 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:29:22.031 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:29:22.031 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:29:22.032 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:29:22.032 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:29:25.109 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:29:25.145 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:29:25.762 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:31:25.888 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:31:25.929 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:31:26.559 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:31:26.559 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:31:26.560 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:31:26.560 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:31:26.560 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:31:26.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:31:29.622 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:31:29.661 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:31:30.263 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:33:30.410 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:33:30.453 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:33:31.091 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:33:31.092 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:33:31.092 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:33:31.092 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:33:31.093 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:33:31.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:33:34.166 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:33:34.206 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:33:34.816 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:35:34.932 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:35:34.971 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:35:35.589 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:35:35.590 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:35:35.590 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:35:35.590 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:35:35.590 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:35:35.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:35:38.645 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:35:38.683 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:35:39.296 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:37:39.414 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:37:39.454 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:37:40.081 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:37:40.081 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:37:40.081 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:37:40.082 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:37:40.082 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:37:40.082 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:37:43.157 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:37:43.195 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:37:43.873 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:39:44.018 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:39:44.094 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:39:44.816 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:39:44.817 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:39:44.817 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:39:44.818 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:39:44.818 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:39:44.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:39:47.893 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:39:47.931 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:39:48.546 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:41:48.693 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:41:48.736 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:41:49.396 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:41:49.397 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:41:49.397 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:41:49.397 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:41:49.398 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:41:49.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:41:52.472 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:41:52.513 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:41:53.122 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:43:53.171 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:43:53.215 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:43:53.861 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:43:53.862 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:43:53.862 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:43:53.862 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:43:53.862 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:43:53.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:43:56.917 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:43:56.957 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:43:57.565 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:45:57.716 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:45:57.757 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:45:58.377 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:45:58.377 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:45:58.378 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:45:58.378 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:45:58.378 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:45:58.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:46:01.429 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:46:01.465 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:46:02.066 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:48:02.166 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:48:02.214 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:48:02.849 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:48:02.850 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:48:02.850 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:48:02.850 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:48:02.850 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:48:02.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:48:05.910 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:48:05.950 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:48:06.560 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:50:06.710 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:50:06.752 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:50:07.383 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:50:07.383 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:50:07.383 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:50:07.384 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:50:07.384 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:50:07.385 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:50:10.454 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:50:10.494 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:50:11.120 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:52:11.267 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:52:11.312 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:52:11.946 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:52:11.946 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:52:11.947 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:52:11.947 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:52:11.947 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:52:11.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:52:15.029 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:52:15.070 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:52:15.689 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:54:15.750 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:54:15.790 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:54:16.427 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:54:16.427 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:54:16.428 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:54:16.428 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:54:16.428 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:54:16.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:54:19.509 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:54:19.550 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:54:20.202 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:56:20.356 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:56:20.408 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:56:21.049 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:56:21.050 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:56:21.050 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:56:21.050 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:56:21.050 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:56:21.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:56:24.117 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:56:24.154 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:56:24.775 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:58:24.867 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:58:24.905 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:58:25.536 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T07:58:25.536 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T07:58:25.536 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T07:58:25.537 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T07:58:25.537 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:58:25.537 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T07:58:28.597 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T07:58:28.634 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T07:58:29.223 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:00:29.318 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:00:29.359 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:00:30.026 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:00:30.026 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:00:30.027 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:00:30.027 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:00:30.027 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:00:30.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:00:33.109 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:00:33.146 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:00:33.743 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:02:33.885 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:02:33.924 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:02:34.586 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:02:34.587 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:02:34.587 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:02:34.588 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:02:34.588 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:02:34.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:02:37.653 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:02:37.690 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:02:38.287 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:04:38.421 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:04:38.461 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:04:39.085 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:04:39.086 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:04:39.086 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:04:39.086 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:04:39.086 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:04:39.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:04:42.165 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:04:42.206 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:04:42.829 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:06:42.971 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:06:43.009 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:06:43.625 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:06:43.625 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:06:43.625 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:06:43.626 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:06:43.626 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:06:43.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:06:46.678 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:06:46.718 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:06:47.333 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:08:47.480 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:08:47.522 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:08:48.151 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:08:48.152 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:08:48.152 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:08:48.152 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:08:48.153 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:08:48.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:08:51.221 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:08:51.263 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:08:51.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:10:51.963 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:10:52.014 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:10:52.654 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:10:52.654 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:10:52.655 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:10:52.655 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:10:52.655 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:10:52.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:10:55.733 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:10:55.774 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:10:56.386 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:12:56.477 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:12:56.518 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:12:57.151 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:12:57.151 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:12:57.152 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:12:57.152 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:12:57.152 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:12:57.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:13:00.213 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:13:00.254 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:13:00.869 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:15:01.014 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:15:01.054 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:15:01.696 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:15:01.697 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:15:01.697 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:15:01.697 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:15:01.697 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:15:01.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:15:04.758 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:15:04.799 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:15:05.403 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:17:05.490 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:17:05.531 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:17:06.164 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:17:06.164 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:17:06.164 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:17:06.165 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:17:06.165 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:17:06.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:17:09.238 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:17:09.278 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:17:09.889 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:19:09.973 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:19:10.014 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:19:10.631 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:19:10.632 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:19:10.632 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:19:10.632 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:19:10.633 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:19:10.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:19:13.686 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:19:13.726 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:19:14.341 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:21:14.453 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:21:14.491 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:21:15.111 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:21:15.112 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:21:15.112 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:21:15.112 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:21:15.113 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:21:15.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:21:18.165 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:21:18.206 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:21:18.818 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:23:19.000 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:23:19.042 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:23:19.681 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:23:19.681 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:23:19.682 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:23:19.682 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:23:19.682 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:23:19.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:23:22.741 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:23:22.780 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:23:23.395 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:25:23.509 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:25:23.585 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:25:24.206 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:25:24.207 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:25:24.207 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:25:24.207 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:25:24.207 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:25:24.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:25:27.285 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:25:27.325 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:25:27.943 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:27:28.038 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:27:28.079 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:27:28.722 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:27:28.723 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:27:28.723 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:27:28.723 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:27:28.723 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:27:28.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:27:31.798 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:27:31.838 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:27:32.448 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:29:32.581 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:29:32.624 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:29:33.274 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:29:33.275 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:29:33.275 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:29:33.275 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:29:33.276 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:29:33.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:29:36.341 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:29:36.381 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:29:36.996 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:31:37.145 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:31:37.186 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:31:37.808 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:31:37.809 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:31:37.809 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:31:37.809 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:31:37.810 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:31:37.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:31:40.885 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:31:40.925 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:31:41.538 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:33:41.686 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:33:41.734 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:33:42.370 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:33:42.371 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:33:42.371 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:33:42.371 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:33:42.371 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:33:42.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:33:45.430 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:33:45.467 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:33:46.109 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:35:46.218 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:35:46.313 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:35:46.940 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:35:46.941 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:35:46.941 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:35:46.941 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:35:46.942 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:35:46.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:35:50.005 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:35:50.042 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:35:50.649 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:37:50.778 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:37:50.819 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:37:51.451 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:37:51.452 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:37:51.452 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:37:51.452 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:37:51.452 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:37:51.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:37:54.518 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:37:54.558 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:37:55.176 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:39:55.318 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:39:55.370 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:39:56.002 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:39:56.002 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:39:56.003 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:39:56.003 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:39:56.003 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:39:56.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:39:59.061 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:39:59.098 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:39:59.705 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:41:59.840 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:41:59.880 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:42:00.512 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:42:00.512 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:42:00.512 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:42:00.513 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:42:00.513 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:42:00.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:42:03.573 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:42:03.613 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:42:04.213 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:44:04.356 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:44:04.396 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:44:05.036 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:44:05.036 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:44:05.037 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:44:05.037 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:44:05.037 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:44:05.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:44:08.117 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:44:08.153 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:44:08.750 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:46:08.831 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:46:08.875 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:46:09.508 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:46:09.508 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:46:09.509 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:46:09.509 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:46:09.509 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:46:09.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:46:12.565 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:46:12.606 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:46:13.226 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:48:13.300 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:48:13.341 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:48:14.009 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:48:14.009 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:48:14.010 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:48:14.010 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:48:14.010 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:48:14.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:48:17.077 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:48:17.115 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:48:17.712 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:50:17.835 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:50:17.904 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:50:18.561 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:50:18.561 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:50:18.562 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:50:18.562 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:50:18.562 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:50:18.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:50:21.622 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:50:21.662 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:50:22.262 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:52:22.356 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:52:22.401 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:52:23.036 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:52:23.037 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:52:23.037 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:52:23.037 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:52:23.037 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:52:23.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:52:26.101 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:52:26.141 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:52:26.752 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:54:26.885 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:54:26.930 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:54:27.587 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:54:27.587 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:54:27.587 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:54:27.588 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:54:27.588 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:54:27.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:54:30.645 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:54:30.685 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:54:31.295 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:56:31.393 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:56:31.432 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:56:32.055 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:56:32.055 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:56:32.056 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:56:32.056 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:56:32.056 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:56:32.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:56:35.129 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:56:35.167 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:56:35.771 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:58:35.871 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:58:35.908 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:58:36.593 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T08:58:36.594 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T08:58:36.594 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T08:58:36.594 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T08:58:36.594 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:58:36.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T08:58:39.669 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T08:58:39.707 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T08:58:40.336 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T09:00:40.482 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T09:00:40.522 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T09:00:41.162 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T09:00:41.163 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T09:00:41.163 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T09:00:41.163 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T09:00:41.164 INFO:teuthology.misc:Compressing logs... 2023-01-03T09:00:41.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T09:00:44.245 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T09:00:44.282 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T09:00:44.895 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T09:02:45.046 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T09:02:45.088 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T09:02:45.726 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T09:02:45.727 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T09:02:45.727 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T09:02:45.727 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T09:02:45.728 INFO:teuthology.misc:Compressing logs... 2023-01-03T09:02:45.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T09:02:48.790 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T09:02:48.831 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T09:02:49.490 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T09:04:49.588 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T09:04:49.628 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T09:04:50.299 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T09:04:50.300 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T09:04:50.300 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T09:04:50.300 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T09:04:50.300 INFO:teuthology.misc:Compressing logs... 2023-01-03T09:04:50.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T09:04:53.365 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T09:04:53.403 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T09:04:54.019 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T09:06:54.146 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T09:06:54.186 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T09:06:54.813 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T09:06:54.813 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0'] 2023-01-03T09:06:55.017 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['osd.1'] 2023-01-03T09:06:55.017 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['osd.2'] 2023-01-03T09:06:55.018 INFO:teuthology.misc:Compressing logs... 2023-01-03T09:06:55.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-01-03T09:06:58.069 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi124 2023-01-03T09:06:58.308 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T09:06:58.899 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T09:08:59.022 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T09:08:59.062 INFO:teuthology.kill:Killing Pids: {13982} 2023-01-03T09:08:59.694 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0'] 2023-01-03T09:08:59.695 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['osd.0']