2023-01-02T16:16:57.608 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2023-01-02T16:16:57.742 INFO:teuthology.lock.ops:Start node 'smithi018.front.sepia.ceph.com' reimaging 2023-01-02T16:16:57.743 INFO:teuthology.lock.ops:Updating [smithi018.front.sepia.ceph.com]: reset os type and version on server 2023-01-02T16:16:57.744 INFO:teuthology.lock.ops:Updating smithi018.front.sepia.ceph.com on lock server 2023-01-02T16:16:57.759 INFO:teuthology.lock.ops:Node 'smithi018.front.sepia.ceph.com' reimaging is complete 2023-01-02T16:16:57.760 INFO:teuthology.lock.ops:Start node 'smithi037.front.sepia.ceph.com' reimaging 2023-01-02T16:16:57.760 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: reset os type and version on server 2023-01-02T16:16:57.761 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2023-01-02T16:16:57.776 INFO:teuthology.lock.ops:Node 'smithi037.front.sepia.ceph.com' reimaging is complete 2023-01-02T16:16:57.776 INFO:teuthology.lock.ops:Start node 'smithi080.front.sepia.ceph.com' reimaging 2023-01-02T16:16:57.776 INFO:teuthology.lock.ops:Updating [smithi080.front.sepia.ceph.com]: reset os type and version on server 2023-01-02T16:16:57.777 INFO:teuthology.lock.ops:Updating smithi080.front.sepia.ceph.com on lock server 2023-01-02T16:16:57.791 INFO:teuthology.lock.ops:Node 'smithi080.front.sepia.ceph.com' reimaging is complete 2023-01-02T16:16:57.905 INFO:teuthology.provision.fog.smithi018:Scheduling deploy of ubuntu 22.04 2023-01-02T16:16:57.913 INFO:teuthology.provision.fog.smithi037:Scheduling deploy of ubuntu 22.04 2023-01-02T16:16:57.928 INFO:teuthology.provision.fog.smithi080:Scheduling deploy of ubuntu 22.04 2023-01-02T16:16:58.159 INFO:teuthology.orchestra.console:Power off smithi018 2023-01-02T16:16:58.171 INFO:teuthology.orchestra.console:Power off smithi037 2023-01-02T16:16:58.181 INFO:teuthology.orchestra.console:Power off smithi080 2023-01-02T16:16:58.310 INFO:teuthology.orchestra.console:Power off for smithi018 completed 2023-01-02T16:16:58.315 INFO:teuthology.orchestra.console:Power off for smithi037 completed 2023-01-02T16:16:58.320 INFO:teuthology.orchestra.console:Power off for smithi080 completed 2023-01-02T16:16:58.412 INFO:teuthology.orchestra.console:Power on smithi018 2023-01-02T16:16:58.418 INFO:teuthology.orchestra.console:Power on smithi037 2023-01-02T16:16:58.425 INFO:teuthology.orchestra.console:Power on smithi080 2023-01-02T16:17:02.676 INFO:teuthology.orchestra.console:Power on for smithi080 completed 2023-01-02T16:17:02.777 INFO:teuthology.provision.fog.smithi080:Waiting for deploy to finish 2023-01-02T16:17:06.778 INFO:teuthology.orchestra.console:Power on for smithi018 completed 2023-01-02T16:17:06.783 INFO:teuthology.orchestra.console:Power on for smithi037 completed 2023-01-02T16:17:06.879 INFO:teuthology.provision.fog.smithi018:Waiting for deploy to finish 2023-01-02T16:17:06.884 INFO:teuthology.provision.fog.smithi037:Waiting for deploy to finish 2023-01-02T16:25:12.313 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.80 2023-01-02T16:25:21.397 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.80 2023-01-02T16:25:30.453 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.80 2023-01-02T16:25:31.257 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37 2023-01-02T16:25:39.509 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.80 2023-01-02T16:25:40.309 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37 2023-01-02T16:25:49.365 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37 2023-01-02T16:26:26.805 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37 2023-01-02T16:26:32.809 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37 2023-01-02T16:26:38.814 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37 2023-01-02T16:26:44.818 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37 2023-01-02T16:26:45.518 WARNING:teuthology.provision.fog:timed out 2023-01-02T16:26:51.035 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-02T16:26:51.496 INFO:teuthology.provision.fog.smithi037:Node is ready 2023-01-02T16:26:51.550 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037.front.sepia.ceph.com 2023-01-02T16:26:51.597 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37 smithi037.front.sepia.ceph.com smithi037 2023-01-02T16:26:51.744 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-02T16:26:51.907 INFO:teuthology.provision.fog.smithi037:Deploy complete! 2023-01-02T16:26:52.176 INFO:teuthology.provision.fog.smithi080:Node is ready 2023-01-02T16:26:52.222 INFO:teuthology.orchestra.run.smithi080.stdout:smithi080.front.sepia.ceph.com 2023-01-02T16:26:52.271 INFO:teuthology.orchestra.run.smithi080.stdout:172.21.15.80 smithi080.front.sepia.ceph.com smithi080 2023-01-02T16:26:52.562 INFO:teuthology.provision.fog.smithi080:Deploy complete! 2023-01-02T16:27:59.289 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-02T16:29:00.293 WARNING:teuthology.provision.fog:timed out 2023-01-02T16:29:24.725 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.18 2023-01-02T16:29:31.733 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.18 2023-01-02T16:29:37.735 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.18 2023-01-02T16:29:43.739 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.18 2023-01-02T16:29:52.794 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.18 2023-01-02T16:29:58.998 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-02T16:29:59.636 INFO:teuthology.provision.fog.smithi018:Node is ready 2023-01-02T16:29:59.685 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018.front.sepia.ceph.com 2023-01-02T16:29:59.734 INFO:teuthology.orchestra.run.smithi018.stdout:172.21.15.18 smithi018.front.sepia.ceph.com smithi018 2023-01-02T16:30:00.069 INFO:teuthology.provision.fog.smithi018:Deploy complete! 2023-01-02T16:30:00.283 INFO:teuthology.lock.ops:Checking smithi018.front.sepia.ceph.com 2023-01-02T16:30:00.284 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-02T16:30:00.299 INFO:teuthology.lock.ops:Checking smithi037.front.sepia.ceph.com 2023-01-02T16:30:00.299 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-02T16:30:00.312 INFO:teuthology.lock.ops:Checking smithi080.front.sepia.ceph.com 2023-01-02T16:30:00.312 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-02T16:30:00.326 INFO:teuthology.lock.ops:Updating [smithi018.front.sepia.ceph.com]: set os type and version on server 2023-01-02T16:30:00.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-02T16:30:00.549 INFO:teuthology.orchestra.run.smithi018.stdout:x86_64 2023-01-02T16:30:00.597 INFO:teuthology.orchestra.run.smithi018.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-01-02T16:30:00.598 INFO:teuthology.orchestra.run.smithi018.stdout:NAME="Ubuntu" 2023-01-02T16:30:00.598 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION_ID="22.04" 2023-01-02T16:30:00.598 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-01-02T16:30:00.599 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION_CODENAME=jammy 2023-01-02T16:30:00.599 INFO:teuthology.orchestra.run.smithi018.stdout:ID=ubuntu 2023-01-02T16:30:00.599 INFO:teuthology.orchestra.run.smithi018.stdout:ID_LIKE=debian 2023-01-02T16:30:00.599 INFO:teuthology.orchestra.run.smithi018.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-02T16:30:00.600 INFO:teuthology.orchestra.run.smithi018.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-02T16:30:00.600 INFO:teuthology.orchestra.run.smithi018.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-02T16:30:00.600 INFO:teuthology.orchestra.run.smithi018.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-02T16:30:00.600 INFO:teuthology.orchestra.run.smithi018.stdout:UBUNTU_CODENAME=jammy 2023-01-02T16:30:00.601 INFO:teuthology.lock.ops:Updating smithi018.front.sepia.ceph.com on lock server 2023-01-02T16:30:00.617 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: set os type and version on server 2023-01-02T16:30:00.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-02T16:30:00.971 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2023-01-02T16:30:01.020 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-01-02T16:30:01.020 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="Ubuntu" 2023-01-02T16:30:01.020 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="22.04" 2023-01-02T16:30:01.020 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-01-02T16:30:01.021 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_CODENAME=jammy 2023-01-02T16:30:01.021 INFO:teuthology.orchestra.run.smithi037.stdout:ID=ubuntu 2023-01-02T16:30:01.021 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE=debian 2023-01-02T16:30:01.021 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-02T16:30:01.022 INFO:teuthology.orchestra.run.smithi037.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-02T16:30:01.022 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-02T16:30:01.022 INFO:teuthology.orchestra.run.smithi037.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-02T16:30:01.023 INFO:teuthology.orchestra.run.smithi037.stdout:UBUNTU_CODENAME=jammy 2023-01-02T16:30:01.023 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2023-01-02T16:30:01.038 INFO:teuthology.lock.ops:Updating [smithi080.front.sepia.ceph.com]: set os type and version on server 2023-01-02T16:30:01.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-02T16:30:01.387 INFO:teuthology.orchestra.run.smithi080.stdout:x86_64 2023-01-02T16:30:01.435 INFO:teuthology.orchestra.run.smithi080.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-01-02T16:30:01.436 INFO:teuthology.orchestra.run.smithi080.stdout:NAME="Ubuntu" 2023-01-02T16:30:01.436 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION_ID="22.04" 2023-01-02T16:30:01.436 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-01-02T16:30:01.436 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION_CODENAME=jammy 2023-01-02T16:30:01.437 INFO:teuthology.orchestra.run.smithi080.stdout:ID=ubuntu 2023-01-02T16:30:01.437 INFO:teuthology.orchestra.run.smithi080.stdout:ID_LIKE=debian 2023-01-02T16:30:01.437 INFO:teuthology.orchestra.run.smithi080.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-02T16:30:01.437 INFO:teuthology.orchestra.run.smithi080.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-02T16:30:01.437 INFO:teuthology.orchestra.run.smithi080.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-02T16:30:01.438 INFO:teuthology.orchestra.run.smithi080.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-02T16:30:01.438 INFO:teuthology.orchestra.run.smithi080.stdout:UBUNTU_CODENAME=jammy 2023-01-02T16:30:01.438 INFO:teuthology.lock.ops:Updating smithi080.front.sepia.ceph.com on lock server 2023-01-02T16:30:01.527 INFO:teuthology.dispatcher.supervisor:Running job 7131031 2023-01-02T16:30:01.528 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-01-02_16:15:57-krbd-pacific_16.2.11_RC4-distro-default-smithi/7131031 --name yuriw-2023-01-02_16:15:57-krbd-pacific_16.2.11_RC4-distro-default-smithi --description krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/no-resets} -- /home/teuthworker/archive/yuriw-2023-01-02_16:15:57-krbd-pacific_16.2.11_RC4-distro-default-smithi/7131031/orig.config.yaml 2023-01-02T16:30:01.530 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-01-02_16:15:57-krbd-pacific_16.2.11_RC4-distro-default-smithi/7131031 2023-01-02T16:30:01.531 INFO:teuthology.dispatcher.supervisor:Job PID: 19123 2023-01-02T16:30:01.531 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-01-03T04:30:18.685 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T04:30:19.269 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:30:19.910 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T04:30:19.910 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T04:30:19.910 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T04:30:19.911 INFO:teuthology.misc:Compressing logs... 2023-01-03T04:30:19.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T04:30:20.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T04:30:20.182 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:30:20.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T04:30:20.437 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:30:38.967 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T04:30:39.012 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:30:39.638 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T04:32:39.683 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T04:32:39.725 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:32:40.364 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T04:32:40.364 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T04:32:40.364 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T04:32:40.365 INFO:teuthology.misc:Compressing logs... 2023-01-03T04:32:40.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T04:32:40.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T04:32:40.633 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:32:40.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T04:32:40.886 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:32:43.926 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T04:32:43.965 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:32:44.579 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T04:34:44.629 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T04:34:44.670 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:34:45.342 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T04:34:45.343 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T04:34:45.343 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T04:34:45.343 INFO:teuthology.misc:Compressing logs... 2023-01-03T04:34:45.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T04:34:45.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T04:34:45.623 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:34:45.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T04:34:45.887 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:34:48.949 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T04:34:48.990 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:34:49.604 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T04:36:49.654 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T04:36:49.714 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:36:50.426 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T04:36:50.427 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T04:36:50.427 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T04:36:50.427 INFO:teuthology.misc:Compressing logs... 2023-01-03T04:36:50.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T04:36:50.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T04:36:50.707 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:36:50.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T04:36:50.968 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:36:54.006 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T04:36:54.045 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:36:54.645 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T04:38:54.695 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T04:38:54.737 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:38:55.375 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T04:38:55.375 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T04:38:55.376 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T04:38:55.376 INFO:teuthology.misc:Compressing logs... 2023-01-03T04:38:55.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T04:38:55.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T04:38:55.646 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:38:55.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T04:38:55.900 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:38:58.965 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T04:38:59.006 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:38:59.664 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T04:40:59.719 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T04:40:59.762 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:41:00.396 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T04:41:00.396 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T04:41:00.396 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T04:41:00.397 INFO:teuthology.misc:Compressing logs... 2023-01-03T04:41:00.397 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T04:41:00.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T04:41:00.675 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:41:00.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T04:41:00.929 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:41:03.989 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T04:41:04.029 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:41:04.676 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T04:43:04.725 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T04:43:04.765 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:43:05.406 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T04:43:05.406 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T04:43:05.407 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T04:43:05.407 INFO:teuthology.misc:Compressing logs... 2023-01-03T04:43:05.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T04:43:05.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T04:43:05.727 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:43:05.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T04:43:05.981 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:43:09.045 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T04:43:09.082 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:43:09.687 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T04:45:09.735 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T04:45:09.774 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:45:10.402 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T04:45:10.403 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T04:45:10.403 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T04:45:10.403 INFO:teuthology.misc:Compressing logs... 2023-01-03T04:45:10.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T04:45:10.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T04:45:10.671 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:45:10.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T04:45:10.928 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:45:13.974 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T04:45:14.017 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:45:14.623 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T04:47:14.672 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T04:47:14.713 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:47:15.381 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T04:47:15.382 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T04:47:15.382 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T04:47:15.382 INFO:teuthology.misc:Compressing logs... 2023-01-03T04:47:15.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T04:47:15.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T04:47:15.695 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:47:15.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T04:47:15.955 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:47:18.997 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T04:47:19.039 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:47:19.701 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T04:49:19.749 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T04:49:19.786 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:49:20.436 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T04:49:20.436 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T04:49:20.436 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T04:49:20.437 INFO:teuthology.misc:Compressing logs... 2023-01-03T04:49:20.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T04:49:20.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T04:49:20.710 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:49:20.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T04:49:20.965 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:49:24.021 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T04:49:24.097 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:49:24.700 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T04:51:24.747 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T04:51:24.790 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:51:25.453 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T04:51:25.454 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T04:51:25.454 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T04:51:25.454 INFO:teuthology.misc:Compressing logs... 2023-01-03T04:51:25.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T04:51:25.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T04:51:25.739 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:51:25.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T04:51:26.005 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:51:29.046 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T04:51:29.087 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:51:29.701 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T04:53:29.746 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T04:53:29.788 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:53:30.430 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T04:53:30.431 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T04:53:30.431 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T04:53:30.431 INFO:teuthology.misc:Compressing logs... 2023-01-03T04:53:30.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T04:53:30.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T04:53:30.751 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:53:30.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T04:53:31.011 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:53:34.069 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T04:53:34.110 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:53:34.724 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T04:55:34.774 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T04:55:34.814 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:55:35.446 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T04:55:35.447 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T04:55:35.447 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T04:55:35.447 INFO:teuthology.misc:Compressing logs... 2023-01-03T04:55:35.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T04:55:35.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T04:55:35.719 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:55:35.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T04:55:35.979 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:55:39.030 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T04:55:39.070 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:55:39.715 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T04:57:39.761 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T04:57:39.801 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:57:40.466 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T04:57:40.466 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T04:57:40.466 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T04:57:40.467 INFO:teuthology.misc:Compressing logs... 2023-01-03T04:57:40.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T04:57:40.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T04:57:40.740 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:57:40.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T04:57:41.001 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:57:44.053 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T04:57:44.095 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:57:44.706 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T04:59:44.754 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T04:59:44.792 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:59:45.448 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T04:59:45.449 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T04:59:45.449 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T04:59:45.449 INFO:teuthology.misc:Compressing logs... 2023-01-03T04:59:45.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T04:59:45.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T04:59:45.721 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:59:45.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T04:59:45.978 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T04:59:49.046 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T04:59:49.086 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T04:59:49.687 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:01:49.731 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:01:49.772 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:01:50.414 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:01:50.414 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:01:50.415 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:01:50.415 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:01:50.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:01:50.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:01:50.690 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:01:50.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:01:50.967 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:01:54.005 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T05:01:54.047 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:01:54.658 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:03:54.704 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:03:54.745 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:03:55.367 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:03:55.368 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:03:55.368 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:03:55.368 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:03:55.368 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:03:55.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:03:55.642 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:03:55.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:03:55.898 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:03:58.965 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T05:03:59.002 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:03:59.613 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:05:59.661 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:05:59.703 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:06:00.328 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:06:00.329 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:06:00.329 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:06:00.329 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:06:00.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:06:00.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:06:00.647 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:06:00.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:06:00.901 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:06:03.958 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T05:06:03.999 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:06:04.614 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:08:04.660 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:08:04.703 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:08:05.323 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:08:05.324 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:08:05.324 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:08:05.324 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:08:05.325 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:08:05.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:08:05.650 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:08:05.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:08:05.906 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:08:08.952 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T05:08:08.993 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:08:09.624 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:10:09.671 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:10:09.711 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:10:10.346 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:10:10.347 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:10:10.347 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:10:10.347 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:10:10.348 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:10:10.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:10:10.625 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:10:10.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:10:10.881 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:10:13.945 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T05:10:13.986 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:10:14.598 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:12:14.682 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:12:14.725 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:12:15.410 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:12:15.411 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:12:15.411 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:12:15.411 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:12:15.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:12:33.909 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi018 2023-01-03T05:12:33.951 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:12:34.573 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:14:34.617 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:14:34.658 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:14:35.298 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:14:35.298 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:14:35.299 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:14:35.299 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:14:35.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:14:35.410 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:14:36.523 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:14:37.635 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:14:38.748 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:14:39.858 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:14:40.967 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:14:42.078 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:14:43.191 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:14:44.302 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:14:45.413 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:14:45.413 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi018 2023-01-03T05:14:45.450 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:14:46.074 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:16:46.127 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:16:46.168 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:16:46.805 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:16:46.806 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:16:46.806 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:16:46.807 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:16:46.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:16:46.918 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:16:48.028 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:16:49.138 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:16:50.249 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:16:51.361 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:16:52.473 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:16:53.586 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:16:54.696 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:16:55.766 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:16:56.875 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:16:56.876 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi018 2023-01-03T05:16:56.913 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:16:57.520 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:18:57.573 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:18:57.646 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:18:58.302 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:18:58.303 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:18:58.303 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:18:58.303 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:18:58.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:18:58.377 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:18:59.447 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:19:00.517 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:19:01.589 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:19:02.700 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:19:03.815 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:19:04.928 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:19:06.000 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:19:07.112 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:19:08.222 ERROR:teuthology.orchestra.connection:Error authenticating with smithi018.front.sepia.ceph.com: Authentication failed. 2023-01-03T05:19:08.223 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi018 2023-01-03T05:19:08.260 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:19:08.889 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:21:08.937 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:21:08.979 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:21:09.712 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:21:09.713 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:21:09.713 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:21:09.713 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:21:09.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:22:09.717 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi018 2023-01-03T05:22:09.784 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:22:10.540 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:24:10.593 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:24:10.640 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:24:11.388 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:24:11.389 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:24:11.389 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:24:11.389 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:24:11.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:24:11.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:24:11.856 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:24:12.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:24:12.102 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:24:12.283 INFO:teuthology.orchestra.run.smithi018.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-03T05:24:12.284 INFO:teuthology.orchestra.run.smithi018.stderr:tar: Error is not recoverable: exiting now 2023-01-03T05:24:12.313 INFO:teuthology.orchestra.run.smithi037.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-03T05:24:12.313 INFO:teuthology.orchestra.run.smithi037.stderr:tar: Error is not recoverable: exiting now 2023-01-03T05:24:12.407 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:24:13.138 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:26:13.183 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:26:13.227 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:26:13.976 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:26:13.977 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:26:13.977 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:26:13.977 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:26:13.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:26:14.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:26:14.172 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:26:14.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:26:14.473 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:26:14.671 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T05:26:14.672 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T05:26:14.716 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:26:15.446 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:28:15.494 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:28:15.548 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:28:16.294 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:28:16.295 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:28:16.295 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:28:16.295 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:28:16.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:28:16.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:28:16.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:28:16.750 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:28:16.938 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T05:28:16.940 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T05:28:16.984 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:28:17.708 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:30:17.758 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:30:17.800 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:30:18.502 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:30:18.503 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:30:18.503 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:30:18.503 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:30:18.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:30:18.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:30:18.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:30:18.963 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:30:19.160 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T05:30:19.161 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T05:30:19.205 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:30:19.914 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:32:19.959 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:32:19.996 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:32:20.635 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:32:20.635 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:32:20.636 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:32:20.636 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:32:20.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:32:20.809 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:32:21.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:32:21.125 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:32:21.332 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T05:32:21.334 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T05:32:21.371 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:32:22.001 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:34:22.046 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:34:22.086 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:34:22.805 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:34:22.806 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:34:22.806 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:34:22.806 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:34:22.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:34:22.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:34:23.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:34:23.278 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:34:23.478 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T05:34:23.479 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T05:34:23.516 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:34:24.161 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:36:24.212 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:36:24.253 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:36:24.988 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:36:24.989 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:36:24.989 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:36:24.989 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:36:24.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:36:25.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:36:25.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:36:25.426 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:36:25.612 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T05:36:25.613 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T05:36:25.656 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:36:26.415 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:38:26.465 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:38:26.507 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:38:27.198 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:38:27.199 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:38:27.199 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:38:27.200 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:38:27.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:38:27.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:38:27.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:38:27.653 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:38:27.862 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T05:38:27.863 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T05:38:27.905 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:38:28.718 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:40:28.773 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:40:28.814 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:40:29.536 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:40:29.536 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:40:29.537 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:40:29.537 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:40:29.538 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:40:29.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:40:30.044 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:40:30.058 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:40:30.269 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T05:40:30.270 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T05:40:30.310 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:40:31.007 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:42:31.054 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:42:31.092 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:42:31.753 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:42:31.754 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:42:31.754 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:42:31.754 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:42:31.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:42:31.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:42:32.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:42:32.199 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:42:32.397 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T05:42:32.398 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T05:42:32.436 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:42:33.054 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:44:33.103 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:44:33.140 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:44:33.800 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:44:33.801 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:44:33.801 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:44:33.802 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:44:33.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:44:33.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:44:34.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:44:34.238 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:44:34.445 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T05:44:34.447 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T05:44:34.488 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:44:35.129 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:46:35.176 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:46:35.218 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:46:35.849 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:46:35.850 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:46:35.850 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:46:35.850 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:46:35.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:46:36.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:46:36.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:46:36.271 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:46:36.459 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T05:46:36.460 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T05:46:36.501 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:46:37.137 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:48:37.186 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:48:37.223 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:48:37.848 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:48:37.848 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:48:37.849 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:48:37.849 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:48:37.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:48:38.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:48:38.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:48:38.285 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:48:38.476 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T05:48:38.478 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T05:48:38.514 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:48:39.128 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:50:39.173 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:50:39.211 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:50:39.830 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:50:39.831 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:50:39.831 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:50:39.831 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:50:39.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:50:40.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:50:40.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:50:40.265 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:50:40.476 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T05:50:40.478 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T05:50:40.517 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:50:41.144 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:52:41.191 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:52:41.228 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:52:41.863 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:52:41.864 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:52:41.864 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:52:41.865 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:52:41.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:52:42.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:52:42.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:52:42.284 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:52:42.489 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T05:52:42.491 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T05:52:42.527 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:52:43.119 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:54:43.166 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:54:43.207 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:54:43.847 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:54:43.848 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:54:43.848 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:54:43.848 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:54:43.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:54:44.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:54:44.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:54:44.274 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:54:44.474 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T05:54:44.475 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T05:54:44.514 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:54:45.151 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:56:45.198 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:56:45.234 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:56:45.852 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:56:45.852 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:56:45.853 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:56:45.853 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:56:45.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:56:46.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:56:46.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:56:46.330 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:56:46.537 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T05:56:46.539 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T05:56:46.576 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:56:47.217 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T05:58:47.264 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T05:58:47.305 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:58:47.930 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T05:58:47.931 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T05:58:47.931 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T05:58:47.931 INFO:teuthology.misc:Compressing logs... 2023-01-03T05:58:47.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T05:58:48.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T05:58:48.373 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T05:58:48.387 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T05:58:48.585 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T05:58:48.586 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T05:58:48.626 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T05:58:49.243 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:00:49.286 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:00:49.323 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:00:49.961 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:00:49.961 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:00:49.961 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:00:49.962 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:00:49.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:00:50.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:00:50.251 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:00:50.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:00:50.511 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:00:50.701 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T06:00:50.702 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T06:00:50.743 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:00:51.354 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:02:51.400 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:02:51.438 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:02:52.065 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:02:52.065 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:02:52.066 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:02:52.066 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:02:52.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:02:52.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:02:52.382 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:02:52.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:02:52.633 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:02:52.838 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T06:02:52.840 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T06:02:52.878 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:02:53.484 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:04:53.527 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:04:53.569 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:04:54.201 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:04:54.201 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:04:54.202 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:04:54.202 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:04:54.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:04:54.461 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:04:54.475 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:04:54.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:04:54.766 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:04:54.969 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T06:04:54.970 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T06:04:55.005 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:04:55.622 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:06:55.671 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:06:55.710 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:06:56.366 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:06:56.366 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:06:56.367 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:06:56.367 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:06:56.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:06:56.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:06:56.643 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:06:56.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:06:56.904 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:06:57.097 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T06:06:57.099 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T06:06:57.137 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:06:57.783 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:08:57.833 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:08:57.872 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:08:58.502 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:08:58.502 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:08:58.503 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:08:58.503 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:08:58.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:08:58.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:08:58.778 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:08:59.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:08:59.040 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:08:59.217 INFO:teuthology.orchestra.run.smithi080.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-03T06:08:59.219 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-03T06:08:59.257 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:08:59.911 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:10:59.959 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:10:59.999 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:11:00.633 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:11:00.634 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:11:00.634 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:11:00.634 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:11:00.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:11:00.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:11:00.903 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:11:01.149 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:11:01.164 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:11:19.541 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:11:19.583 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:11:20.199 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:13:20.263 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:13:20.299 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:13:20.929 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:13:20.930 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:13:20.930 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:13:20.930 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:13:20.930 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:13:21.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:13:21.205 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:13:21.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:13:21.507 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:13:24.565 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:13:24.602 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:13:25.194 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:15:25.245 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:15:25.286 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:15:25.902 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:15:25.903 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:15:25.903 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:15:25.903 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:15:25.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:15:26.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:15:26.183 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:15:26.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:15:26.438 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:15:29.494 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:15:29.534 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:15:30.146 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:17:30.194 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:17:30.235 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:17:30.860 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:17:30.861 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:17:30.861 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:17:30.861 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:17:30.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:17:31.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:17:31.176 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:17:31.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:17:31.435 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:17:34.485 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:17:34.527 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:17:35.191 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:19:35.244 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:19:35.284 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:19:35.941 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:19:35.942 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:19:35.942 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:19:35.942 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:19:35.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:19:36.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:19:36.227 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:19:36.469 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:19:36.483 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:19:39.541 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:19:39.578 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:19:40.183 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:21:40.232 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:21:40.273 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:21:40.898 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:21:40.898 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:21:40.898 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:21:40.899 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:21:40.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:21:41.201 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:21:41.215 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:21:41.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:21:41.470 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:21:44.533 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:21:44.575 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:21:45.175 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:23:45.220 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:23:45.258 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:23:45.872 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:23:45.873 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:23:45.873 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:23:45.873 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:23:45.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:23:46.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:23:46.147 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:23:46.433 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:23:46.447 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:23:49.494 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:23:49.569 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:23:50.175 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:25:50.225 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:25:50.268 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:25:50.964 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:25:50.965 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:25:50.965 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:25:50.965 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:25:50.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:25:51.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:25:51.239 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:25:51.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:25:51.539 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:25:54.581 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:25:54.624 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:25:55.293 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:27:55.341 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:27:55.387 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:27:56.093 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:27:56.093 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:27:56.094 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:27:56.094 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:27:56.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:27:56.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:27:56.413 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:27:56.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:27:56.664 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:27:59.734 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:27:59.778 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:28:00.453 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:30:00.524 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:30:00.565 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:30:01.321 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:30:01.322 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:30:01.322 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:30:01.322 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:30:01.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:30:01.637 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:30:01.651 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:30:01.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:30:01.902 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:30:04.950 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:30:04.993 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:30:05.698 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:32:05.748 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:32:05.790 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:32:06.490 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:32:06.491 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:32:06.491 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:32:06.492 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:32:06.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:32:06.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:32:06.767 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:32:07.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:32:07.033 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:32:10.101 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:32:10.145 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:32:10.838 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:34:10.888 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:34:10.933 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:34:11.654 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:34:11.655 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:34:11.655 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:34:11.655 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:34:11.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:34:11.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:34:11.935 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:34:12.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:34:12.193 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:34:15.253 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:34:15.297 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:34:15.991 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:36:16.039 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:36:16.082 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:36:16.766 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:36:16.767 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:36:16.767 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:36:16.767 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:36:16.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:36:17.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:36:17.055 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:36:17.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:36:17.306 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:36:20.373 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:36:20.416 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:36:21.086 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:38:21.138 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:38:21.180 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:38:21.882 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:38:21.882 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:38:21.883 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:38:21.883 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:38:21.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:38:22.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:38:22.153 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:38:22.393 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:38:22.406 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:38:25.462 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:38:25.504 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:38:26.215 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:40:26.265 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:40:26.308 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:40:27.019 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:40:27.020 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:40:27.020 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:40:27.020 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:40:27.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:40:27.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:40:27.307 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:40:27.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:40:27.561 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:40:30.613 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:40:30.657 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:40:31.303 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:42:31.350 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:42:31.392 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:42:32.097 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:42:32.098 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:42:32.098 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:42:32.098 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:42:32.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:42:32.369 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:42:32.381 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:42:32.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:42:32.647 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:42:35.701 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:42:35.744 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:42:36.438 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:44:36.486 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:44:36.526 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:44:37.235 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:44:37.235 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:44:37.235 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:44:37.236 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:44:37.236 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:44:37.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:44:37.518 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:44:37.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:44:37.776 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:44:40.821 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:44:40.867 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:44:41.590 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:46:41.637 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:46:41.679 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:46:42.366 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:46:42.366 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:46:42.367 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:46:42.367 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:46:42.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:46:42.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:46:42.653 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:46:42.900 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:46:42.914 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:46:45.973 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:46:46.016 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:46:46.693 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:48:46.741 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:48:46.787 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:48:47.491 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:48:47.491 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:48:47.491 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:48:47.492 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:48:47.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:48:47.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:48:47.762 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:48:47.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:48:48.011 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:48:51.061 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:48:51.133 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:48:51.810 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:50:51.858 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:50:51.903 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:50:52.634 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:50:52.635 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:50:52.635 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:50:52.635 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:50:52.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:50:52.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:50:52.910 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:50:53.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:50:53.158 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:50:56.213 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:50:56.257 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:50:56.939 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:52:56.989 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:52:57.030 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:52:57.720 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:52:57.721 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:52:57.721 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:52:57.721 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:52:57.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:52:57.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:52:57.994 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:52:58.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:52:58.253 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:53:01.301 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:53:01.346 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:53:02.051 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:55:02.105 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:55:02.148 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:55:02.847 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:55:02.848 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:55:02.848 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:55:02.848 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:55:02.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:55:03.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:55:03.129 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:55:03.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:55:03.386 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:55:06.453 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:55:06.496 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:55:07.151 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:57:07.203 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:57:07.243 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:57:07.903 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:57:07.904 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:57:07.904 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:57:07.904 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:57:07.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:57:08.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:57:08.179 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:57:08.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:57:08.434 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:57:11.477 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:57:11.514 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:57:12.144 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T06:59:12.197 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T06:59:12.238 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:59:12.904 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T06:59:12.904 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T06:59:12.905 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T06:59:12.905 INFO:teuthology.misc:Compressing logs... 2023-01-03T06:59:12.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T06:59:13.172 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T06:59:13.186 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:59:13.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T06:59:13.445 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T06:59:16.501 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T06:59:16.542 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T06:59:17.142 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:01:17.192 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:01:17.228 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:01:17.871 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:01:17.871 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:01:17.871 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:01:17.872 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:01:17.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:01:18.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:01:18.143 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:01:18.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:01:18.403 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:01:21.461 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:01:21.502 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:01:22.132 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:03:22.179 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:03:22.219 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:03:22.852 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:03:22.852 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:03:22.852 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:03:22.853 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:03:22.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:03:23.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:03:23.166 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:03:23.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:03:23.426 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:03:26.485 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:03:26.521 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:03:27.137 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:05:27.192 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:05:27.228 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:05:27.866 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:05:27.867 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:05:27.867 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:05:27.867 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:05:27.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:05:28.125 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:05:28.138 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:05:28.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:05:28.396 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:05:31.445 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:05:31.482 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:05:32.067 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:07:32.114 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:07:32.149 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:07:32.755 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:07:32.756 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:07:32.756 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:07:32.756 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:07:32.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:07:33.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:07:33.026 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:07:33.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:07:33.286 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:07:36.341 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:07:36.380 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:07:36.973 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:09:37.023 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:09:37.064 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:09:37.708 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:09:37.709 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:09:37.709 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:09:37.710 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:09:37.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:09:37.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:09:37.983 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:09:38.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:09:38.243 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:09:41.301 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:09:41.339 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:09:41.971 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:11:42.022 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:11:42.063 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:11:42.700 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:11:42.700 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:11:42.700 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:11:42.701 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:11:42.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:11:42.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:11:42.968 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:11:43.213 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:11:43.224 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:11:46.293 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:11:46.332 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:11:46.928 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:13:46.974 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:13:47.014 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:13:47.642 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:13:47.642 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:13:47.642 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:13:47.643 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:13:47.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:13:47.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:13:47.923 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:13:48.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:13:48.187 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:13:51.253 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:13:51.312 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:13:51.914 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:15:51.962 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:15:52.003 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:15:52.619 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:15:52.620 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:15:52.620 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:15:52.620 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:15:52.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:15:52.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:15:52.890 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:15:53.132 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:15:53.145 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:15:56.181 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:15:56.222 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:15:56.841 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:17:56.888 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:17:56.927 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:17:57.565 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:17:57.566 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:17:57.566 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:17:57.566 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:17:57.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:17:57.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:17:57.848 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:17:58.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:17:58.110 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:18:01.176 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:18:01.220 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:18:01.838 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:20:01.883 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:20:01.925 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:20:02.571 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:20:02.571 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:20:02.572 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:20:02.572 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:20:02.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:20:02.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:20:02.833 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:20:03.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:20:03.086 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:20:06.133 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:20:06.172 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:20:06.781 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:22:06.829 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:22:06.866 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:22:07.526 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:22:07.527 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:22:07.527 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:22:07.527 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:22:07.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:22:07.784 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:22:07.798 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:22:08.044 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:22:08.058 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:22:11.125 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:22:11.165 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:22:11.779 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:24:11.827 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:24:11.863 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:24:12.491 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:24:12.492 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:24:12.492 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:24:12.492 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:24:12.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:24:12.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:24:12.806 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:24:13.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:24:13.118 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:24:16.181 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:24:16.219 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:24:16.819 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:26:16.865 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:26:16.904 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:26:17.529 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:26:17.529 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:26:17.530 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:26:17.530 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:26:17.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:26:17.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:26:17.809 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:26:18.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:26:18.105 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:26:21.173 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:26:21.213 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:26:21.820 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:28:21.864 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:28:21.905 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:28:22.549 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:28:22.550 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:28:22.550 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:28:22.550 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:28:22.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:28:22.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:28:22.831 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:28:23.069 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:28:23.083 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:28:26.133 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:28:26.170 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:28:26.767 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:30:26.815 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:30:26.865 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:30:27.503 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:30:27.504 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:30:27.504 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:30:27.504 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:30:27.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:30:27.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:30:27.813 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:30:28.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:30:28.117 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:30:31.157 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:30:31.195 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:30:31.822 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:32:31.888 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:32:31.924 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:32:32.553 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:32:32.554 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:32:32.554 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:32:32.554 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:32:32.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:32:32.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:32:32.873 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:32:33.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:32:33.125 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:32:36.182 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:32:36.222 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:32:36.827 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:34:36.882 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:34:36.923 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:34:37.548 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:34:37.549 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:34:37.549 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:34:37.549 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:34:37.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:34:37.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:34:37.825 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:34:38.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:34:38.079 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:34:41.141 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:34:41.182 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:34:41.789 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:36:41.835 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:36:41.873 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:36:42.494 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:36:42.495 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:36:42.495 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:36:42.495 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:36:42.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:36:42.763 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:36:42.780 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:36:43.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:36:43.041 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:36:46.101 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:36:46.144 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:36:46.758 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:38:46.802 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:38:46.839 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:38:47.464 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:38:47.466 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:38:47.466 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:38:47.466 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:38:47.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:38:47.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:38:47.785 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:38:48.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:38:48.037 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:38:51.093 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:38:51.189 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:38:51.812 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:40:51.857 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:40:51.898 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:40:52.533 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:40:52.534 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:40:52.534 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:40:52.534 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:40:52.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:40:52.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:40:52.815 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:40:53.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:40:53.080 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:40:56.149 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:40:56.187 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:40:56.791 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:42:56.838 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:42:56.875 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:42:57.513 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:42:57.514 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:42:57.514 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:42:57.514 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:42:57.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:42:57.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:42:57.846 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:42:58.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:42:58.095 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:43:01.142 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:43:01.183 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:43:01.807 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:45:01.852 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:45:01.890 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:45:02.545 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:45:02.546 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:45:02.546 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:45:02.546 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:45:02.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:45:02.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:45:02.826 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:45:03.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:45:03.135 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:45:06.198 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:45:06.240 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:45:06.855 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:47:06.901 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:47:06.939 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:47:07.555 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:47:07.556 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:47:07.556 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:47:07.556 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:47:07.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:47:07.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:47:07.826 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:47:08.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:47:08.086 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:47:11.125 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:47:11.164 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:47:11.772 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:49:11.823 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:49:11.860 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:49:12.479 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:49:12.479 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:49:12.480 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:49:12.480 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:49:12.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:49:12.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:49:12.739 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:49:12.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:49:13.000 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:49:16.053 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:49:16.094 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:49:16.751 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:51:16.802 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:51:16.840 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:51:17.492 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:51:17.493 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:51:17.493 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:51:17.493 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:51:17.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:51:17.746 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:51:17.759 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:51:18.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:51:18.018 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:51:21.077 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:51:21.116 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:51:21.720 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:53:21.766 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:53:21.806 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:53:22.447 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:53:22.447 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:53:22.448 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:53:22.448 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:53:22.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:53:22.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:53:22.720 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:53:22.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:53:22.975 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:53:26.037 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:53:26.076 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:53:26.748 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:55:26.796 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:55:26.833 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:55:27.476 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:55:27.477 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:55:27.477 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:55:27.478 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:55:27.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:55:27.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:55:27.758 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:55:28.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:55:28.014 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:55:31.061 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:55:31.098 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:55:31.701 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:57:31.747 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:57:31.784 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:57:32.419 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:57:32.420 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:57:32.420 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:57:32.420 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:57:32.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:57:32.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:57:32.746 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:57:33.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:57:33.018 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:57:36.085 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:57:36.124 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:57:36.721 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T07:59:36.766 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T07:59:36.802 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:59:37.422 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T07:59:37.422 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T07:59:37.423 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T07:59:37.423 INFO:teuthology.misc:Compressing logs... 2023-01-03T07:59:37.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T07:59:37.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T07:59:37.703 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:59:37.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T07:59:37.967 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T07:59:41.013 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T07:59:41.054 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T07:59:41.660 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:01:41.708 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:01:41.746 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:01:42.379 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:01:42.379 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:01:42.380 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:01:42.380 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:01:42.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:01:42.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:01:42.658 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:01:42.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:01:42.909 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:01:45.973 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:01:46.010 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:01:46.648 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:03:46.696 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:03:46.736 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:03:47.351 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:03:47.351 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:03:47.351 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:03:47.352 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:03:47.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:03:47.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:03:47.663 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:03:47.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:03:47.912 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:03:50.966 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:03:51.047 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:03:51.657 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:05:51.704 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:05:51.744 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:05:52.367 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:05:52.367 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:05:52.368 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:05:52.368 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:05:52.368 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:05:52.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:05:52.642 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:05:52.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:05:52.889 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:05:55.958 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:05:55.995 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:05:56.595 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:07:56.640 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:07:56.680 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:07:57.311 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:07:57.312 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:07:57.312 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:07:57.313 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:07:57.313 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:07:57.609 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:07:57.624 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:07:57.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:07:57.874 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:08:00.917 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:08:00.953 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:08:01.557 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:10:01.602 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:10:01.642 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:10:02.275 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:10:02.275 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:10:02.276 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:10:02.276 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:10:02.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:10:02.537 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:10:02.550 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:10:02.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:10:02.805 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:10:05.845 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:10:05.881 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:10:06.485 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:12:06.529 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:12:06.567 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:12:07.203 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:12:07.204 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:12:07.204 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:12:07.204 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:12:07.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:12:07.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:12:07.519 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:12:07.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:12:07.770 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:12:10.837 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:12:10.878 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:12:11.489 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:14:11.544 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:14:11.584 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:14:12.219 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:14:12.220 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:14:12.220 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:14:12.221 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:14:12.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:14:12.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:14:12.498 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:14:12.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:14:12.794 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:14:15.862 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:14:15.902 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:14:16.513 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:16:16.561 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:16:16.601 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:16:17.230 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:16:17.230 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:16:17.230 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:16:17.231 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:16:17.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:16:17.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:16:17.513 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:16:17.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:16:17.774 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:16:20.821 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:16:20.862 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:16:21.472 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:18:21.525 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:18:21.566 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:18:22.201 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:18:22.201 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:18:22.202 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:18:22.202 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:18:22.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:18:22.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:18:22.467 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:18:22.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:18:22.755 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:18:25.813 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:18:25.853 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:18:26.468 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:20:26.514 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:20:26.554 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:20:27.186 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:20:27.187 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:20:27.187 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:20:27.187 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:20:27.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:20:27.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:20:27.507 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:20:27.801 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:20:27.815 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:20:30.869 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:20:30.907 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:20:31.547 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:22:31.609 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:22:31.649 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:22:32.292 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:22:32.293 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:22:32.293 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:22:32.293 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:22:32.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:22:32.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:22:32.604 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:22:32.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:22:32.900 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:22:35.957 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:22:35.995 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:22:36.599 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:24:36.650 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:24:36.686 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:24:37.298 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:24:37.298 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:24:37.298 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:24:37.299 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:24:37.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:24:37.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:24:37.573 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:24:37.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:24:37.871 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:24:40.917 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:24:40.957 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:24:41.562 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:26:41.606 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:26:41.645 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:26:42.272 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:26:42.272 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:26:42.273 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:26:42.273 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:26:42.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:26:42.532 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:26:42.544 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:26:42.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:26:42.844 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:26:45.909 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:26:45.946 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:26:46.565 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:28:46.608 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:28:46.646 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:28:47.316 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:28:47.317 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:28:47.317 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:28:47.318 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:28:47.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:28:47.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:28:47.591 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:28:47.825 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:28:47.839 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:28:50.901 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:28:50.965 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:28:51.565 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:30:51.613 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:30:51.653 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:30:52.315 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:30:52.315 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:30:52.316 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:30:52.316 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:30:52.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:30:52.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:30:52.597 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:30:52.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:30:52.853 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:30:55.894 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:30:55.936 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:30:56.546 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:32:56.599 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:32:56.638 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:32:57.249 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:32:57.250 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:32:57.250 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:32:57.250 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:32:57.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:32:57.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:32:57.527 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:32:57.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:32:57.777 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:33:00.821 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:33:00.861 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:33:01.493 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:35:01.539 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:35:01.577 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:35:02.195 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:35:02.196 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:35:02.196 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:35:02.196 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:35:02.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:35:02.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:35:02.467 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:35:02.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:35:02.717 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:35:05.781 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:35:05.823 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:35:06.435 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:37:06.481 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:37:06.528 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:37:07.203 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:37:07.203 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:37:07.204 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:37:07.204 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:37:07.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:37:07.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:37:07.470 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:37:07.717 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:37:07.731 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:37:10.773 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:37:10.813 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:37:11.443 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:39:11.487 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:39:11.524 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:39:12.155 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:39:12.155 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:39:12.155 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:39:12.156 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:39:12.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:39:12.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:39:12.433 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:39:12.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:39:12.736 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:39:15.797 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:39:15.836 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:39:16.437 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:41:16.488 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:41:16.524 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:41:17.130 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:41:17.131 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:41:17.131 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:41:17.131 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:41:17.132 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:41:17.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:41:17.402 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:41:17.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:41:17.663 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:41:20.725 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:41:20.762 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:41:21.352 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:43:21.397 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:43:21.433 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:43:22.039 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:43:22.040 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:43:22.040 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:43:22.041 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:43:22.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:43:22.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:43:22.312 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:43:22.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:43:22.574 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:43:25.621 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:43:25.661 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:43:26.263 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:45:26.308 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:45:26.348 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:45:26.976 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:45:26.977 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:45:26.977 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:45:26.977 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:45:26.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:45:27.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:45:27.247 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:45:27.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:45:27.496 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:45:30.550 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:45:30.591 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:45:31.247 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:47:31.299 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:47:31.341 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:47:31.980 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:47:31.980 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:47:31.981 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:47:31.981 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:47:31.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:47:32.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:47:32.251 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:47:32.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:47:32.505 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:47:35.574 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:47:35.617 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:47:36.269 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:49:36.317 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:49:36.368 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:49:36.993 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:49:36.994 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:49:36.994 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:49:36.994 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:49:36.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:49:37.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:49:37.270 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:49:37.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:49:37.531 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:49:40.598 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:49:40.639 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:49:41.241 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:51:41.293 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:51:41.334 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:51:41.952 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:51:41.952 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:51:41.953 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:51:41.953 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:51:41.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:51:42.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:51:42.230 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:51:42.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:51:42.478 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:51:45.530 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:51:45.581 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:51:46.191 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:53:46.252 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:53:46.299 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:53:46.950 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:53:46.951 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:53:46.951 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:53:46.951 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:53:46.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:53:47.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:53:47.229 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:53:47.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:53:47.485 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:53:50.549 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:53:50.628 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:53:51.239 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:55:51.288 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:55:51.324 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:55:51.951 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:55:51.951 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:55:51.951 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:55:51.952 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:55:51.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:55:52.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:55:52.232 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:55:52.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:55:52.487 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:55:55.541 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:55:55.578 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:55:56.183 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T08:57:56.226 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T08:57:56.263 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:57:56.893 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T08:57:56.894 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T08:57:56.894 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T08:57:56.895 INFO:teuthology.misc:Compressing logs... 2023-01-03T08:57:56.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T08:57:57.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T08:57:57.167 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:57:57.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T08:57:57.437 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T08:58:00.501 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T08:58:00.542 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T08:58:01.154 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T09:00:01.202 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T09:00:01.239 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T09:00:01.906 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T09:00:01.907 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T09:00:01.907 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T09:00:01.907 INFO:teuthology.misc:Compressing logs... 2023-01-03T09:00:01.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T09:00:02.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T09:00:02.177 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T09:00:02.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T09:00:02.434 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T09:00:05.493 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T09:00:05.536 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T09:00:06.221 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T09:02:06.273 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T09:02:06.310 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T09:02:06.930 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T09:02:06.931 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T09:02:06.931 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T09:02:06.931 INFO:teuthology.misc:Compressing logs... 2023-01-03T09:02:06.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T09:02:07.189 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T09:02:07.202 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T09:02:07.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T09:02:07.469 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T09:02:10.518 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T09:02:10.559 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T09:02:11.184 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T09:04:11.234 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T09:04:11.272 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T09:04:11.891 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T09:04:11.891 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T09:04:11.892 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T09:04:11.892 INFO:teuthology.misc:Compressing logs... 2023-01-03T09:04:11.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T09:04:12.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T09:04:12.165 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T09:04:12.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T09:04:12.420 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T09:04:15.478 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T09:04:15.518 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T09:04:16.138 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T09:06:16.195 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T09:06:16.236 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T09:06:16.867 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T09:06:16.868 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T09:06:16.868 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T09:06:16.869 INFO:teuthology.misc:Compressing logs... 2023-01-03T09:06:16.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T09:06:17.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T09:06:17.135 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T09:06:17.392 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T09:06:17.404 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T09:06:20.470 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T09:06:20.512 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T09:06:21.135 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T09:08:21.183 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T09:08:21.222 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T09:08:21.847 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T09:08:21.847 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T09:08:21.847 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T09:08:21.848 INFO:teuthology.misc:Compressing logs... 2023-01-03T09:08:21.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T09:08:22.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T09:08:22.121 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T09:08:22.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T09:08:22.374 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T09:08:25.429 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T09:08:25.470 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T09:08:26.083 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T09:10:26.132 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T09:10:26.177 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T09:10:26.807 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T09:10:26.808 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T09:10:26.808 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T09:10:26.809 INFO:teuthology.misc:Compressing logs... 2023-01-03T09:10:26.809 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T09:10:27.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T09:10:27.075 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T09:10:27.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T09:10:27.319 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T09:10:30.357 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T09:10:30.399 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T09:10:31.025 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T09:12:31.072 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T09:12:31.108 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T09:12:31.731 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T09:12:31.732 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T09:12:31.732 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T09:12:31.732 INFO:teuthology.misc:Compressing logs... 2023-01-03T09:12:31.733 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T09:12:31.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T09:12:32.004 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T09:12:32.249 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T09:12:32.264 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T09:12:35.317 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T09:12:35.359 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T09:12:35.971 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T09:14:36.020 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T09:14:36.056 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T09:14:36.672 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-01-03T09:14:36.672 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-03T09:14:36.672 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['client.0'] 2023-01-03T09:14:36.673 INFO:teuthology.misc:Compressing logs... 2023-01-03T09:14:36.673 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2023-01-03T09:14:36.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2023-01-03T09:14:36.950 INFO:teuthology.orchestra.run.smithi018.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T09:14:37.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2023-01-03T09:14:37.207 INFO:teuthology.orchestra.run.smithi037.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-03T09:14:40.246 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi080 2023-01-03T09:14:40.287 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T09:14:40.909 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-03T09:16:40.958 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-03T09:16:41.000 INFO:teuthology.kill:Killing Pids: {19123} 2023-01-03T09:16:41.648 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']